oocoun iisiei 



10 156^406 . 

AOIBOS 

IXILS 

IISflTOTlOH 

SPOKS AGERCI 
FOB ,OATt 
»OTB 

Af&ILABU FBOH 

EBBS PBlCE 
OESCBIPIOBS 



010 M8 



ABSTBACT/ 



Cosby, Arthur G.^ Coi(#; i€therill, 6« Bicbard, 

A Synthesis: B€SOQrc€8 in Ival«atioii tot Bnxal 
Pevalopaent* SB]>C Sjatbeaia Serias t2« 
Soatbara Baral Devalofaaat Caatax,, Stata Collega, 
Bias* 4 

Baral Oe?alopaant Sara ica -(COA) , eaabinstoHf 0«C« 
Apr 78 • 
66p« . 

Soatbam Boral Devalopaaat Cantar# Box 5406, 
Bisaiaaippi stata, Biaaiaaippi 3S762 (I1«C0) 

BF-$0«83 BC-$3«l0 Bloa foataga. ' 
Concept Foraation; Btaica; Bva^laation Criteria; 
^Evalaatioa Betboda; «ivalaatit>n Maada; Intcraation 
Batriaval;^In£oraaticn Soaicaa; ^Bodala; Prograa . 
Evaluation ; ntural Davalcpaant; ^Stata of tba Art 
Beviava; Sy^thaaii 



A series of aix papertf tbia syntbaaia of ^iteratoi^e ;^ 



OQ rara/L developaent and evalaation offara a starting pcirt for roral 
davelopaent plai^ning and projacta froa tba national to tbe local 
laval.rxbe corrant state of knowledge ia aaaasa^d and application 
t6cbaiao€ts and aetbbda are pinpointed. In tie introduction, operation 

Sf tbe' Functional' Betvork of tbe aponaoring Soutbern Baral 
avelopaent Center (SBBC) ia exaained« An ovarviav paper cn **fbe 
Process of Bvaluation in Baral Develop aent** includea aby and bov 
evaluating ia carried otft, pitfalls, and a stata-^of-tke<»artv report* 
Betbodological issues confronting tbe avaluator are discuaaed in 
**C<mcept and PbiloaopJiy of Bvaluation and Bjcral Sevelopaent,**^ and tbe 
neat paper, "^Alternative Bodela of Ivala^tion and fbair Application 
to Boral Developaent,** ayataaatically ratiaia II evaluation aodels* 
Issues dt objectivity, integrity, privacy, peracnal bare, ' 
confidentiality, and accessibility are covered in ''Beapcnaibilitiea 
of tbe Evaluation Besearcbers Xbe ASA Cede of Etbica aa a Guideline*" 
Particalar attention to coaputerixed retrieval aystaaa is given in 
**Soarcea of Inforaation for Evaluating Baral Otvelopaant.'' A auaaary 
paper, **EvalaatioA and Buxal Developaent,** toucbea on evaluation 
aodela, evaluation inforaation, needed evaluation training, and an 
SBOC pxograa design/evaluation aorkabof* (BS) 



aaaaaaaaaaaaeaaaaaaaaaa*aaaaaaaaaaaaaaaaaaaaaaaaaaaaa,aaaa««aaaaaa^aaaaa 
fie prod octibna supplied bj BOBS are tbe beat tbat can be aade * 
^ ^ froa tbe original docuaent. * 

eaaaeaaaaaaaaaaa«aaaaaaaaaeaaeaaMiaaaaaaaaaa#aaa«aaaaaaaaaaaaaaaaaaaaaa 



o 



U $,OEPARTMENTOF HEALTH 
EOuCi^lON 4 WELFARE 
NATIONAL INSTITUTE OF 
EDUCATION 

^Mis ooc'jvent m&s seen repro- 

OKEO EXACTLY AS RECE'VBD FROW 
THE PEPSON on ORGANiZATtON ORIGIN- 
AL NGiT PO'NTSOP VIEAOR OPINIONS 
SEATED 00 NOT NECESSARILY REPRE 
SENTOPfiClAL NATIONAL iNSTfTuTEOF 
EDUCATION POS'TiON 0» POUCY 



PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GpANTED BY 




□ 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER 'ERl.O AND 
USERS OF THEERICSYS^TEM ' 



SYNTHESIS 



Resources in Evaluation 
for Rural Development 



00 

© 

CD 
O 



ERIC 




Rural Development Series No. 
Southern Rural Development Centei 

4 

2 



* poSeword- 

• • - - -. •• ^ . _ . 

Under the sponsorship of the Southern Rural' Dev^opment Center 

ten t«am8 of: researchers and educators throughout the soutiiem region 

have devoted the past year to d synthesis of timely an4 practical 

research In selected areas of interest. ^ ^ 

' These ten Functional Networks, each uiiidej the leadership of a 

Center Associate, have prepared larger annotated bibliographies of ^ 

important citations uncovered in their Investigations. These synthesis 

papers follow the bibliographies, and are intended to relate the ^s^£^l 

applications to be. deriv^ from their survey of the literature^ 

Hor^ than just summary documents or reports, these synthesis papers 

can serve as a starting point for rural development planning and projects 

from the national to the local level. They assess the current state of 

knowledge and pinpont techniques and methods for application of these 

findings. / ' ' 

This paper was prepared by the Network on Resources in Evaluation 

■J 

for Rural Development under the leadership of Dr^ Arthur^t?* Cosby and 
Dr. G. Richard Wetherill at Texas A&M University. The Network's . ' 

bibliography and additional copies of this paper are available from the 
Southern Rural Development Center. ' ' 



William H. Linder 
Dire^ctor 

Soutljgrn Rural Development Center 



ASyntbMls* 

RESOURciES IN EVALUATION FOR RURAL ISVELOPMENT 



Dr. Arthur 6. Cosby and Df » 6. Richard VatherlU 
' - Texaa A&M l»ilyeraity 

Co-Chalrperaona 
SRDC Fimctlonal Netverrk 



/ ^ 



Dean Barrett 
Lawrence Clijiton 
Harold Crawford 
Myma Hosklns , 
Howai;d Ladewlg 
Freddie Richards * 
Andres Stelnaetz 
Gerald C. Wheelock 



of Oklahoaa State University 

East Texas State University , 
Evaluator-at-Large ^ • 

Texas University ^ 
' North Carolina State Uniyerslty 
Ptalrle View A&M University 
University of Virginia 
Alabaaa A&M University 



er|c • 



This synthesis Is the product of an SRDC Functlbn§l Network with 
chairmanship ar Texas AfiM University. A contractual agreement ^ 
between the SRDOf Texas A&M University » and the 'Cooperative State 
Research Service of the U. S« Department of Agriculture provided for 
this participation and cooperation* « v 



One of a series of syntheses prepared by research Functional Networks 
for the Southern Rurai Development Center » Mississippi State» 
Mississippi, 



SRDC Synthesis Series #2 
$1.00 

4 ^ 



. * ' TABUS OF COHTKHTS ^ 

Foreword ^ . 

Preface / ^ 

Intfo4uctlqn - Dr. Arthur G. Cosby • . . . . . t • . . 1 

I. The Process of EvaltiAtlon in Sural Developnent r ^ 

Dr. G. Richard JWetherill • ^ . 3 

II* ^The Concept and Philosophy of Evaluation in ^ 

Rural Development - Dr. Howard Ladevig . . ; 13 

III* Alternative Modes of Evaluation and Their 
Application to Rural Deyelopnent ^ 

l5r. G. Richard Wetherill and Jo^n Buttran 22 

IV., Responsibilities of the Evaluation Researcher:^ 
The A.S.A. Code of Ethica ad a Guideline - 
Dr. Lawrence Clinton. 34 

V. Sources of Information for Evaluating Rural 

Development: An Overview - Laura Grayburn, 

Marilyn Magee, and Myrt^ S. Hoskins 43 

VI* Evaluation ^h4-ilural Development r- Dr. Arthur G. 

Cosby and Dr/O. Richard Wetherill 51 



1 



ERIC 




Preface 

^ — ■ 

This manuscript; ,#as prepared as a r^ponse ttk the rural develo^^t 
practitioner's need ^or knowledge on the subject' of evaluaHve reseatch. 
However,. th%^ paper may 'best be vlew^ as just the top of the evaluation 
Iceberg. We hope to have Included nost\of the relevant concerns for 
evaluation from the practitioner's stand^int. 

The contents o% this document are as ^ried as the wide-ranging 

discipliiie of evaluation research. Beginning with the Introduction, 

Coshy presents the concepts of evaluation and bf the "Functional Network" 

which inspired this work. The first paper, "Thi Process of Evaluation 

In Rural Development," by Wetherlll, is an overview of the. field of 

evaluation as it applies- to rural development. Iriclud^.here are such 

topics as: What, Why, and How We Evaluate, Problem^ and Pitfall's in " 

Evaluation, and a State-of-the-Art report. NarrowlnV the scope ^, 

somewhat, "The Concept and Philosophy of Evaluation a\d Rural Develop^ 

ment" by Ladewig, discusses the concept of evaliiation with special ' 

attention directed toward methodological issues confronting the, evaluato^. 

Staying within the methodological realm of evaluation for the momeat, •\ 

the next paper., "Alternative Modes of Evaluation and Their Application \ 

to Rural Development," by Wetherlll and'Bi^tram,. systematically reviews \ 

twenty-pne models that, have been advanced as approaches to doing evaluation. 

c • 
This, systematic synthesis results in the base-phase concept as a way 

of viewing evaluation procedures. 

A great concern for all program evaluators and those to whom 
'evaluation is done'ls'th^t of ethics In evaluation. In ^•Responsibilities 
V of the Evaluation Researcher: The^.A. Code of Ethics As A Guideline," 



Clinton focuses in a nov^ way upon the ethical problems associated with 
the conduct of evaluation. The special responsibilities of evaluation 
researchers Ae outlined along with Issues of objectivity. Integrity, . 
privacy, personal ham, confidentiality, and accessibility. 

At one time or another, both beginning and v6teran evaluators ^ 
will have a need for better ways of accessing useful background naterials 
for their evaluation efforts. "Sources of Informatldn for Evaluating Rural 
Develbpnent: An Overvlew,."l?y Grayba^i;n, Magee, and Hoskins Is an account 
ol the authors' experiences In searching for evaluation and rutal development 
literature. The use and acces'slblllty^of jtonputeiflzed retrieval systons Is 
given particular attention. ' «' • 

For the final paper in this set, "Evaluation and Rural Developio^t," 
by Cosby and Vetherill, the total experience of the Functional Network is 

t • 

capsuli«ed# This paper sunnaarizes the types and varieties of evaluation 
nodels, how infomation pn Jeyaluatlon may be obtained, the type of 
evaluation training that is needed in rural development, and a brief ^* 
description of a program design and evaluation workshop conducted for 
personnel in rural develo^pment . ' 

True synthesis Is a difficult concept to* pin down. This is / 
especially true for two fields as far flung as. those of evaluation 
and rural development. Hopefully, we have included in this document • 
"iOme of the most relevant concerns relating to the juncture of. both . , 
fields. ^ — ^ 

G.R.W, ' L 

§tarkvllle, Mississippi 
January, 1978 



INTRODUCTKW* 



A growing concern at bbth federal and state govdrnment levels is 
for program acccfuntabillty and evaluation. This concern extends to 
rural development. Accountability and evalixatipn are becominjg key issues 
to social scientists performing either as program dj^velopers or evaluators. 
It can be argued that the area of rural >6evelopment has^<speclal difficulties^ 
in addressing these concerns. 

V There has been no systematic attempt to assemble and synthesize 
evaluations related to either previous or xmgoing rural development. 
Likewise, there has been no agreement or, to our knowledge, any systematic 
effort to identify those methods, techniques, and measures which would 
, be of high utility to the ^valuator of rural development programs. Sinc^ 
there appears to be a serious gap with regards to rural development' 
evaluation, and since evaluation, is becoming a'^iequirement for many 
programs that use federal funds,' a bibliographic search and syntheisis of 
relevant literature on rural development has become mandatory. 

The Functional Network, "A Synthesis of Evaluative Res^rch Literature 
tor Rural Development," is a creation of the Southern Rural T)eveloptocnt 
Center in cooperation with tMe Texas Agricultural Experiment Station. 
Its charge has been to address the question of evaluation and nural 
development. Beginning in June 1975, the project was established as 
one of several Functional^ Networks funded by the SRDC to review and synthesize 
literature on a series oF topics critical to the conduct of rural development 
in the southern region. Since the beginning of the Network, its goals have 
stressed the development of a set of annotated bibliogrlphies and a synthesis 
df existing evaluation research literature. It has searched for those 
principles and procedures which hold promise for rural development evaluation 
in terms bf existing programs and the planning and e^aluatloti of future 
programs. Special attention has been given to those sources which are 
relate^ non-economic evaluations - for example, evaluations of programs ' 
designed to raise the overall quality of ^ life,' level of living, life 
satisfactions, knowledge, and leadershlp/decislon-maklng skills of rural 
residents. * ^ ' 



iis introduction was written by Arthur G. Cosby, Associate Professor 
of Rural Sociology at Texas AiM! University currently serving as Center 
Associate for the Southern Rural Development Center. 
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The Functional Network 

% 111 

The Southern Rurai Developmwit Center adopted the Functional Network 
as a mechanism to involve professional^ across the region In review and 
synthesis of extant research literature salient to rural development. 
The Networks generally addressed the question, "What does, the body of ; 
literature tell us about rural development Bad how can we use this Information 
to Improve rural development programs?^' The guidelines ?of the network 
concept allowed for the inclusion of meibbers from a wide range and variety 
of Institutions in the region. This Fujactional Network, , "A Synthesis 
of Evaluation Research Literature for Rural Development," was developed 
in the spirit of open membership and attempted to bring together scholars 
with divergent views and perspectives. The ten members of the Network 
included experiment station researchers and Eattension agents, researchers from 
non-land-grant univer^sities, a federal research administrator, graduate student 
and an evaluator from a private firm. •> • 

\ • ' 
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Introduction 



The rather brief history of evaluation In rural •development has been 
characterized by an attitude of foreboding. This seeto to b^e one of the 
most prevalent problems In this Mea at thf present time. Such an 
attitude appears to steb malkly from a lack <jf Information regarding vhat^ 
principles of evaluation are applicable to rural development. The under- 
girding purpose of this chapter is that of providing rural development 
practitioners with a brief sketch of the tools n^essary to conduct an 
evaluation of rural development programs. Four 'key Issues will be dealt 
with here: U) the purpose of evaluation, (2^ what we evaluate, (3) the 
evaluation process, and (A) the problems and pitfalls in evaluation. 
Although these four key Issuea cannot by any stretch of the imagination / 
be considered as the total range of the evaluation process, they may be 
viewed as ^our issues which any potential evalxiator must addres? before 
getting started in evaluative research. 

I There afre many definitions of evaluation ^ Much of the controversy 
in the field today concerns the development of an adequate definition 
for evaluation [13]. Many of these controversies spring from deflnltlotis * 
of evaluation which lend a threatening connotation to evaluation. Perhaps 
what would be most beneficial for the evaluation of rural development 
programs would be a definition of evaluation which is non-threatening 
in naturi. One. frame in which duch an evaluation definition could be 
couched is one which results from the definitional dichotomy of "evaluation 
for program improvement" versus "evaluation for program justification." 
The most non-threat jning definitions of evaluation tend to come from 
those wKb define evaluation as being for the primary purpose of t>rogram 
improvement, whUe the more threatenlAg definitions relate directly to 
Justification of programs. Therefore, fo^: ouf purposed here, evaluation 
may be defined as the systematip examination of a program in operation 
for the purpose of iiq)rovement [29]. <This process takes the form of 
assessing a. program in terms of its stated goals and objectives. Such 
evaluative information is then used In order to make decisions concerning 
program improvement. Although such a definition may be limited and 
rather elementary, it can serve as a useful starting point for beginnina 
an evaluation. ^ , 

Evaluation research may be considered more applied than "pure." 
Even with all things considered, the ranges of evaluation are great. In , 
terms of things' evaluated, the range may run the ga^t from classroom/teach 
ing-type evaluations, through detailed program eviiluatlons, all the way 



♦IJis article by G. Richard Wetherill, Sociologist, D. S. 
Porest^Sirvlce, Southern Porest Experiment Station, is reprinted from 
Rural Sociolorty in th^ South: 1?77. Haraha N. Mookherjee (Ed.), CookevlUe. 
Tenn.: Tennessee Technological University, 1977. ^ i^itevixxe. 



up to the evaluation of soclaV^bsystems.* In terms of th^ degiree of 
evaluative rigor^ the range may run from cerebral type of evaluations 
through formal evaluations. The techniques of ^valuation research also* 
encompass a vide range, including informal techniques (participant^ 
pbservations or other unobtrusive measures) up to and including Experi- 
mental and quasi-experimental designs. It is usually up to d>e evaluator 
and/or program staff and administrators to determine where in these 
ranges the evaluation may be specif led » Such things must be considered 
in developing a personal philosophy of evaluation. 



^Why Evaluate? The Purpose of Evaluation » 

Everyone evaluates. Our daily lives are filled with decisions we ^ 
must malce based on the conclusions which we draw ^rom the data that we 
receive. This infon^al evaluation, more often^han not non-verbalized, 
is part a:nd parcel of our dally *llving« 

In the case of project or program evaluation^ ^11 too often informal 
evaluation is the only' type that has been done. Given the increased 
national emphasis for requiring program evaluation [12], what is needed 
now is a formalization of the ofttimes informal prodess of evaluatlpn. 
The benefits of such formalization are potentially tremendous. At the 
very le^st, evaluation may be used -to provide documentation i:onceming . 
the history and/or impacts of a program. Better evaluations will provide 
program decis;Lon makers with the necessary evaluation information in 
orde^to help them make better decisions toward better programs. Iii 
this lAsht, program evaluation can be used for both program improvement, 
and program Justification, and provide for program planning' and policy 
making. The fact remains that,, like it or not, pto^am evaluation is 
fast becoming a mandatory part of program operations [30]. ^ 

W hat Po We Evaluate? » * ' 

. ^ 

The answer to the question posed by the title of this^ section is 
not simply answered by saying, "the program." In evaluation, we must 
be concerned with the components which not only make up the program, 
but also make up the evaltiation. Here, we mean such, things as: program 
objectives or gbals, program personnel interaction, program processes, 
resources, intended audiences, 'evaluation objectl.ves, types and levels 
of evaluation data, judgment making processes evaluator roles and 
responsibilities, and the list could» go on» In order for this "primer" 
to be most effective, it seems that possibly two major issues should be 
addressed. When the question is aaked, ^*Wha£ should we evaluate?", the 
first answer should probably be "tne objectives.'* Being the logical 
starting point for an evaluation, the objectives of a program should be 
adequate descriptors of what the program is all about. . Generally, the 
jargon of evaluation labels program objectives as criteria for 
evaluation [19]. In many evaluation designs, criter% become the 
starting point for itiellneatlng what data are to be collected for the 
evaluation effort. 

y. 
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A major evaluation data concern which we have Identified In the 
realm of social program evaluation Is that relating especially to those 
non-economic variabiles which have a bearing on social programs. Part 
of the legacy of social program evaluation, and evaluation research in 
general, has been the concern for cost-benefit analysis. Needless- to 
say, the knowledge, attitudes, skills, ahd aspirations of program 'target 
apdiences are very difficuj4: to operationalize in terms of cost-benefit 
analyses [18]. Doll^rs-and-cents figures do not necessarily apply to 
importa;nt i>rogr^ considerations such as staff interaction. ' There is 
an^ increasing emphasis in social program evaluation toward the measurement 
of non-economic variables. This issue is exemplified thrcfugh such 
considerations as using attitudes of program recipients as a measure of 
program effectiveness. ^ • ' ^ 

What this all seems to point to is an expanded "bag of tricks?' for 
the* program evaluator. The evaluator must be aware of any type of data 
collection device, no matter what the source, which can aid him in more 
effectively evaluating a program. The evaluator must learn to rely not 
only on data gathered from' detailed survey designs, but must also be 
able to apply toore informal techniques such as participant observation 
[10]. Although evaluation data coming from many sources may tend to get 
involved, the chances become greater that fewer important aspects of the 
program will be neglected in the examination (15, 26]. In short, the 
social, program evaluator must be flexible, and al^ys innovative, in 
conducting an evaluation of a social program (1, 8, 9, 17, 24, 28].. 

The Eiyaluatlon Process 

Just I as there is no single, accepted ^outlliJ^ for the procedure 
known as "the scientific method," the nrocess of evaluation also has mo 
single set -of stepd or procedures to follow in conducting an evalvatibn. 
In the Jargon of evaluation, such processes may be called various names. 
In the literature, "the process of evaluation'* may be Subsumed under 
such labels as models, frameworks, or approaches to evaluation [20]. ^ 
Generally, all- these terms refer to the same thing, the evaluation 
process. Rather than to advocate any one particular model over another, 
we haVe found that there are similarities among the models [29]* 'As 
such; we have identified that the^evaluatlon process encoiqiasses several 
separate steps. 

/Logically, any process, especially the evaluation process, must 
have both a starting point and an ending point. The starting point of 
the evaluation process was identified as beginning somewhere prior to 
the actual iaqilemfentation of an evaluation design. The ending point was 
identified as falling soon after the evaluation exercise was" coi^leted. 
These two points in the process of evaluation were considered important 
enough to be phases in and of themselves. Between these two starting 
and ending phases in the process, we Idefntified three basic phases which 
totaled, represent a five-phase process of evaluation. Overall, the 
process of evaluation as pjresented In the following sketch lists the 
five elements which are more or less coonon to all the evaluation designs 
%rhich we examined. 

. 12 ' 



Program 
Description 

'4^ 




Criteria 
Formulation 



1 



^ Evidence 
Ga^rlng 



Siiiiiiiiiiiiiiiii7iifiiiiiiiiiiiiiiiifti 



^Judgment 
Haklng 



Slssexiilnatlon 



■iiC 



.This process of evaluatlon| Is o£ course very general In appearance. 
The benefits of this generalize^ procedure are found mostly In Its 
fiei^lblllty. With a general fra^e^rk^uch as this, an eyaluator may 
specify within each of the basic phases tihe process of evaluation which 
la- most applicable to the social programming evaluated. The details 
are left up .to the evaluator or the evaluation team. 

/j: vflie Pttogram description phage Is the beginning phase of the 
^^"^^evaluatlon process. During this phase the evaluator describes 
the program, this dotumentatlpn Is an attempt tp put down on 
paper what the program Is all about. This serves to direct the 
evaluator, program ^staff, and appropriate audiences as to the , 
progi^tii^s primary goals and« What Its courses of action have been. 
Generally, this first phase consists of Sj written description of 
the program which serves as a documentatfion of the goals ax£d 
^ operations of the program. 



The criteria formuliatlon phase focuses the evaluation on specific 
aspects of the program. While It Is Important to conslBer the 
objectives of the^ program In setting up these criteria, 1^ ir 
almost Impossible 'to examine systematically all aspetts of the. 
program. What is helpful ^here is to .ideijitify the most critical 
areas of the program* s operation. These critical areas then 
become the focus of the evaluation. Standards, or' acceptable 
levels of perfq|piance, must be developed for each of these 
criteria^ > Again, the obje<;tlves of the program cannot be Ignored 
in the ei^ablishmentuof evaluation criteria. 

The evldence-gatheylng phase concerns itself with the gathering of 



iluatlve data. 
€f^ data to be col£ 
gathered concerning 
^according to the ev 
the preceding phase 
of the specified e 
^ and appropr 
this phase. 
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is phase in the proftess of evaluation specifies 
ted for evaluation evidence. Evidence is 
he performance or operation of . the program 
uation criteria formulated and established in 
Data are collected during this phase on each 
uation criteria. Construction of instruments 
fatistical analyses of collected data occur during 
At this point the evaluation is probably more similar 
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to the traditional research investigation than at any other 
phase in the evaluation process. The skills and knowledge 
required of the evaluator ijfelate to the principles axid aethanlcs 
*of research design, aethodoiogy » and analysis. 

The judgnient-»aaklng phase Is the most liQ>ortant phase In the 
whole evaluation process. Wlt^ut judgnent making there would 
be no evaluation. The process of making judgments Is usually 
left up to the evaluator. As a Starting point for making 
Judgments on the program, . the process of evaluation requires 
that the data° collected for evaluation be co^>ared to the criteria 
established ^f or evaluation and Judgmrats be made on this informa- 
tion regarding the performance of the program. Such judgments 
are usually based on the discrepancy between the ideal conditions 
presented in the criteria and the real conditions as indicated by 
the data collection effort. Judgaients aaid/or program' alternatives 
are based upon the recognition and identification of such 
discrepancies. * 
• 

The dissemination phase is the last phase of the evaluation process. 
' Very few guidelines or procedures for repotting evaluation results 
have been set down. However, such iElndings are usually reported 
back to clients. Sometimes special contractual arrangements are 
made to include wider audiences; The use of the evaluative 
findings depends upon the status of the program operation. ^However, 
it is very important to remember that evaluation results must be 
fed back into the program. Only through such feedback qan an 
evaluation make an impact upon future programming and program 
decisions. 

^ These then are the five basic f>ha8es 04 the evaluation process. 
Although th^e five phases are necessary in any evaluation,, additional 
specifications may easily be made within the phases In order to customize 
the process to any rural development program under examination. It must 
also be noted that these phases, or any specification thereof, do not 
necessarily have to be in a straight linear sequence. In other words, 
the process of evaluation is not locked into a 1, 2, 3, 4, 5 sequence. 
Re-specification^ may have to be made during the process of evaluation. 
When s^h become! the case, it^is necessary to retreat back to an 
earlier phase in the development of the evaluation (dotted arrows, see 
sketch on paige 6) so that the program may be adequately evaluated. 
Re-documentation of the program or respecif ication of the criteria may 
have to Se done, edpecially because of the often d]^namlc nature of 
development projects. 



Problems and Pitfalls 



) 



Needless to say, prograa evaluation is a dangerous thing, fvaluation 
has in its ^ power the fate of the program. Misunderstandings as to th^ 
conduct of evaluation may often' lead to splits in program staff sometimes 
going in as many as three directi<^ns, including admlnistfators, evaluators, 
and field woorkers, all on different sided of 'the issue [11, 23, 31]. 
Unt4J. such a time as the process ojf evaluatl^on is^ more coxn>letely understood 
by all rural development personnel, any evaluator or potential evaluator 
of a rural development project must be aware of the problems and pitfalls 
inherent in the Evaluation'' research enterprise. A brief list of some of 
the things an evaluator should be concerned with in order to avoid 
problems ndght look sonething^like this; ^ \ 

1. The ethical responsibilities of an evaluator should be 
beyond reproach [6]. , 

2. The responsibility for jprogram evaluation should be delected 
, to one person. This would include the commensurate authority 

to best be able to inplem^t evaluation plant [3, 7]* 

3. The total staff of the program should be^f ormed about what 
^ is going on in an evaluation. Valuation plans must be 

communlcateii to all relevant program personnel, so there are 
nos surprfslfs with respect to what is going to happen for the 
evaluation [15, 21]. 

4. Reliance on a single source of dat^ as evidence fox evaluation 
should be scrutinized most closely, having data from many 
different sources usually lends more Validity to evaluation 
evidence [15, 26]. \ 

5. The evaluator is usually treated as an Intruder into the 
program. As sucl^, he must be aware of the pressures and 
constraints on the program b^ing evaluated. The evaluator 
must be a diplomat [4, 14, 28]. 

6. The evaluator, aloi^ i^tk program staff, must get together 
early in the process of evaluation in order to negotiate the 
terms of evaluation. Areas of such negotiation Include: 
ways and means of disseminatd'On, adcess to project personnel 
and records^ audiences for the evaluation results, roles and 
responsibilities of the evalisdtor, etc. [2, 5, 16, 25]. 

The evaluator, as well as the evaluation, must be flexible, 
ually, the dynamic nature of rural development programs 
dttdes a strict reliance on detailed evaluation achedulea: 
to/ a certain degree, evaluation plaqs should not act as a ^ 
ralght-jacket to evaluation [22]. 

Under most circumstances, evaluators ought not to get embroiled ^ 
with actual deeislon-tfking functions In a program. The typical 
tola of the evaluator «lght be that of spelling out options 
for program alternatives but not that of making actual decisions 
exclusive. o£ {Jrogjeam ata££ Ll^l^ 
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9* The evaluator should be cowdtted tp evaluation. Hla attitude 

toward the evaluation process ahou^^pbe positive. This Is In 

" ^^^^--^ contrast to^ his attitude toward ther*prograa, which ahould be 
unbiased [4, 22]. " 

10. Evaluation feedback should be given to appropriate progran 
personnel ad the evaluation progresses. This does not Bean, 
however y that evaluation results aliould b€ given out preautturely 
(15, 211. / 

' Um If possible, g^t sowone to react to your evaluation planfl^ ^ 
' before laq^leaentlng the plana. But know when t:o sto^ aakljig 
for advice [2,, 5, it}. 
%. 

The program evaluator is a pq^tentlally powerful j€£t potentially 
danger oifs role. An evaluator must Vk a specializ^ed generallat. The 
evafluator must have a knowledge of basic reitaarch methods and statistics, 
technique^ ^ of evaluation (the evaluator *8 bag* of tricks), a knowledge of 
program dev^opment technlqi^es (Including proposal writing, budgeting,, 
staffing y program planning , progta implementation, and above all, the 
ability to define goald* and objectives) , and knowledge of the social 
i^stem in which the program is operating as' well as interpersonal dynamics 
internal to the prodram. ^ 

i * / - ' ' ' 

In sum, a program evaluator, as well as program evaluation, is in a 
tmlque' position to aid rural dev^opment. Rural develo^mietit programs which 
have evaluation built into the program not only. have a higher likelihood 
oS success^ bi|t are also good ex&a^les for subsequent development programs. 
There is no magic f ormdla for'^evaluatloHf nor is any program evaluator a 
magician. The fact remains, however, that program ^Valuation has become 
almost sudatory not only f or^ rural development programs , but also f dr 
social action programs in general. , The more that is known about the 
evalxuitlve research enterprise^ the better it will be for ^he acceptance 
of evaluative results^ henclb program improvement. When this- occurs, 
-evaluative research will not be the *^shady*V enterprise which many people 
suggest that it is. ^ % 
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•CHAPTER II 



' THE CONCEPT AMD FHILOSOFHT OF EVALDATIOH 
, ' IN SUBAL DEVELOPMBHT* 

S ' ■ ' . 

Evaluation research Is becoming Increasingly inportant to a large 
nuaber of people for a variety of reasons. Administrators confronted 
with increasing ooiq>etitlon for fm^ing are looking to evaluative research 
td *deterBlne progria effectiveness. Hany socioldgists and other social 
scientists concerned about the failure of .governBental pplicy making to 
be inforaed by the inforaation and knowledge generated in their disciplines 
, regard evaluative research as an avenue for social science input into 
^ govemaental policy asking. Legislators striving for aore efficient utili* 
zation of liid.ted' resources art initiating legislation nhich. requires 
evaluative research for assessaent of whether or not federal policies and 
prograas are aeeting the needs of sc^iety. 



Although Coc^ftds hag passed legislation thit >as included both the 
requireaents fot evaluation and the fundlk&g for carxytig it out, the federal 
govemaent as a whdle an4 aost agencies have no gverall ays tea for effectively 
evaluating program and project ef fc»:tiyeness [15]. Instead, aost evaluations 
have been guided by 'definitions of eviduation ahich have equated it vith either 
a^ureaent and teafeng, ^fulfilling of . objectiv6s« or professional Judgaents [1]* 

In spite of the increased presence of evAuation research, auch 
disagreeaent exists between and among adainlstAtors, t^ro^aa personnel, 
and socia!|^ scientists concerning the appropriate «uid legitimate role of 
evaluation in social action prograas. I^act, serious questions exist 
pertaining to the capacity of evaluation research to provide sufficiently 
sotmd inforaation. - 

- ♦ 

Mann [8], for Example ^ suggests that the qualH^ of eval^tive research 
is reaarkably poor and that there is little difference £n the results of 
eyaluatj^ studies conducted in different content areas. After coaparing 
n^ly 200 evaluation studies of slailar design be concluded, "Specifically, 
there is no indication that the findings ^of evalMitive research are 
influenced by the aethod tested, the con$^t area in which the test is 
> conducted, the change criteria used or the aetbodological qnali^ of the 
^ study of which the evaluation is aade./' 

f 

There is a growing persuasion~particularly in ' th« field of education— 
that Mch of the difficulty encounterfSd in evaloation research stems from 
the use of experimental design to evaluate ongoing progmt or <»perational 
systems. Sucfamsn [13] argues that action and service programs a^e ongoing 
events which cannot be separated from the entire operation process; 
therefore, an evaluation design must be utUised which provides input to 
the. total programming process rather tiian conf Ineas^t to the terainal ^ 
effects of a prograa. Operations research constitutes such a desiffci. 



*Thl« article by HoMrd Udwrlg, Prosrw Braloatioa SiMclallst, 
T«M Agricultural Eictension Sanrlca, Tocaa AM Dnlverait^, la reprinted 
fro« Bural Soclolottr in the South: 1976 . Virginia P. Steelaen (ed.), 
Beton louge, LA: Loulaiana State naiireraity, 1976. 
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J^rtlng from the premise that administrators, lirogram^ personnel, 
and soctJal scientists are equally ijespondlble :^or the conduct of evaluation, 
this chapter will address several major Issues •and some nuances that 
affect the appropriate and legitimate role of evaluation research. Tiie 
Issues Include the environment In vhlc^ evaluation takes place, types of 
evaluation design, and selection of 'an evaluation design for social 
act'lon programs in rural development. 



Environment for Evaluatiojn 

Cuba [4] suggests that any professional area that is so jaich avoided; 
that produces so many anxieties; that immobilizes the very people who 
want to avail themselves of it; that is picapable of operational definition, 
^even by Its most trained advocates (who ib fact render bad advice to the 
practitioners who consult them); which is not effective in answering 
reasonable and iiiq>ortant questions; and which, has made little apparent, 
effort to isolate and ameliorate its most serious problems — must indeed 
give us pause. 

» \ '^^^ " 

At present, a wide chasm exists between the desites of administrators 
and the requirements of researchers over the development of an evaluation 
design.. Because of this chasm participation in evaluation research often 
has proven harmful ta those involved in its undertaking. ^ 

For^ exaiq>le, programs have been develbped which all too often did not 
have clearly specified goals but did have unrealistically high expectations. 
The setting of such goals is a matter of policy, but when the policy makers 
do not specify goals clearly and sti^ require evaluation, the- evalua tors are 
put In the uncomfortable position of deciding what the goals were that the 
policy makers had in mind. Thus, when lj|gh expectations are not realized, 
negative findings are not easily accepted. [10]. . ^ 

Dissatisfaction with evaluation findings may result in evaluation 
> practices being attacked on methodological grounds for political and ^ 
. bureaucratic purposes. Tbe researcher becomes the center of controversy and 
the debate transfers from ideology to methodology [16]. * 

' On the other hand, administrators are keenly aware of poor or ihappro- 
prlate evaluation designs. Often the alma and character of programs are 
mlacomfeiv^ by tbe evaluator and as a result evaluat;lon design and ^ 
execution are^ limited value [3]. Also, ample evidence ex^stis of ' 
evaluations that have focused rather heavily on testing of fayiHDtheses 
of Interest primarily to the evaluator and of little, interest or v^lue 
to the decision maker [5]. 



ERLC 



Under existing conditions it is difficult to. blame program personnel for 
avoiding evaluation when the basic relationship between l^aluator and 
practitioner id one of the ^former Judging the work of the latter. This 
judgment often constitutes a threat to those re9ponsible for program 
activities' reaching stated objectives. 

Correspondingly, social scientists cannot b€ faulted for their 
reluctance to conduct evaluation when they must contend with conceptual 
^aiut methodological problems; problems of relationship, status, and function; 
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practical problems*; and problems of career and revard.% Added to that, 
evaluation is becoming increasingly political [141. • 

Types of Evaluation ^ ^ ^ ' 

^ tfach of the difficulty encountered in evaluation originates in the / 
use of an evaluation design that is not appropriate to the type of program . 
being evaluated. Evaluation can occur on several levels and use dif f etent ^ 
research designs which vary in their approximation to the ideal scientific 
experiment (131', — > 

Within the broad category of reseax;ch the distinguishing pgiri^erist^cs 
of evaluatipn research is that the meaiBurement of outcomes takeS/i^Ice. 
either under ac|udl operating conditions or under conditions that reflect 
in sotoe reasonable degree the problens' associated with Operating actual 
programs [161. Within this bicad category of evaluation research two 
type's of evaluation may be distinguished: those based on experimental 
design and those based on evaluation of the total system. v 

The purpose of field experiment evaluation is. to determine: (1) the 
validity of a particular approach a^ |i means tovard^lie achievement of 
some desired change or objective, and (Zf the ability to institute a workable 
program based on that approach (13]. Systems models place emphasis on 
increasing program effectiveness rather than success or failure. Although 
systems evaluation can serve sueh other functicms as knowledge building 
and theory testing, unless it gains serious hearing when program decisions 
are made, it- fa^s in its major purpose [14 J. . • 

Held Experiments ^ 

. The field expe^i^ent evaluation is based on principles of the experi- 
mental model. Guided by a j^re-determined plan, the Valuator intentionally 
manipulates one oic »pre^ of the independent variables, and then observes 
the changes £n the dependent variables. To insure ^hat observed 
differences mong groups really reflect differences in the independent 
variables, the experimental groups are usually selected rah4omly [11]. 

Although, the lo«ic of experimrat design is nearly Infallible, evaluators 
have Wkd limited success in implementing a true experimentation outside the 
laboratory.' Instead, most studies havf" collected evidence b#sed on 
observation of natural variations in the Independent variables. This type 
of setting is referred to as a ^uasi-eaip»lMntal design. The tn>ortant 
characteristic of quasi--experiments £s that some force clearly unrelated*-' 
to the dependent variable causes the variation in the independent variable 
liij* 

^ Several qtiasl-ezperliMntal design* are presented [7], ranging from 
one. shot case studies" to eiq>crlaental-control group designs with randoalsa- 
tion and rigorous aanagewnt of all foreseeable variables Influencing 
lnternal;«nd cstfemal validity. Each haa appllc^lllty to a particular 

pjroitra*. . The reader aay wish to consult the references listed 
thorough discussion of the Internal and external validity llnltatlons 
each #hradlgB. 
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Systems Models 

The objective of systems evaludtioti Is constant program assessment and 
Improvement. A systems model is programmatic research whose major function 
Is to aid those responsible for making decisions In planning, and adjusting 
their activities in an attempt to Increase the probability of achieving 
desired action or service goals. Such models have been called process 
models and have had their major development In the field of opeijations 
research [13]. Two variations af oi^eratlons research will be examined— 
one whose major orientation is program input and on^ Concerned with program . 
output. . ' ^ L \ ' . ' - 

Product evaluation measures and interprets attainments during the ' 
project term/as well as at theiend of a project cycl^.- Activities Include , 
devislnfe^peratlonal definitions of objectives, measuring' crl.ter la associated 
with the objectives of the activity, cotaparing these Measurements with 
predetermined absolute or relative standards ♦ and making rational^ interpret- 
ations of the outcomes using the recorded content, inpt^t, and pxocess 
information. Experimental design can and should be employed in certain 
kinds of product evaluation situations. However, chaiige in design 
application may be required to account for individual 4lf£erencesNin 
clientele'. ' \ 

.Others provide additional support for this systems orientation to 
evaluation. Weiss [14] says 'that programs have other functions and have 
consequences besides achieving official poals and these are worthy of study 
5uchman [13 1 contends- that experimental design with its femphasls on success- 
failure is neither applicable nor relevant to ongoing prdgrams. Critics 
of experimental design usually pffer three major iudgmentis of its uPility. 

1. Experimental design falls to provide for continual ptogram improvement 

V 

2. Experimental design provides useful information at the terminal point 
of a program but is useless in planning and liiq?lemen ting a program. 

3. Experimental control is not practical in most instances. 

Thus, operations research is the appropriate design because it presents 
the entire process of program development and management with the focus of 
evaluation being Upon day-to-day operation of the system ad a whole. The 
results of this type of evaluation research become applicable to decision 
making at all stagea of program planning, development and operation [13]. 



Systems Output Evaltiatlon ^ 

The function of output evaluation is to provide feedback from results to 
decisions. Output evaluation: (1) assesses the effectiveness of an ingoing , 
program in achieving its objectives, (2) relies on the principles of, research 
design to distinguiah a program's effects from those of other fbrces wprking 
in a situation, tod (3) alms at program improvement through a modification of 
current operations [IS]. ,^ 
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> Output evaluation la concerned with questions of program effectlvcsiess' 
more than of program efficiency. In this respect. It Is goal-oriented, focusing 
on output rather than Input. Output evaluation differs from program analysis 
and policy analysis, which usually .compare^ existing and hypothetical 
alternative program solutions to the s^cme problem. 



This design distinguished four major types of evaluation — program 
l^wct evaluation, program strategy evaluation, prefect evaluation and 
project , ratings 

Progbsm l^iact evaluation Is assessment of the- overall effectiveness 
of a national program In meeting Its objectives, or assessment of the 
relative effectiveness of tvo or more programs In meeting common objective^. 
Its goal Is to assist policy makers In reaching decisions on program funding 
iBvels or on J^osslble r^irectlon of a program. 

Systems Input Ei^aluatlon 

One of the most thorough examples of evaluation of an operational system 
(education) Is provided by the Phi Delta Kappa National Study Committee [9]. 
A brief summary of their efforts will be presented to Illustrate evaluation 
based on Input effectiveness « For a more thorough treatment of educational 
evaluation and decision making, the reader la ref er&ed to the reference 
cited. - ^ ... 

In contrast to research methodologr where the ei^hasls Is on genefatlpn\ 
of new knowledge, evaluation methodology~i>artlcularly In the field of 
educatlon—ls concerned wl^ the process of delineating,^ obtaining and 
providing useful Information for Judging decision alternatives. This 
Information Is not necessarily new knowledge and Is hlg^ily particularistic and 
specific to a decision situation rather than belpg generallzable >tQ macty 
or all settings. ' , ' v^^^ 

Althou^ the term "Judging*^ la central** to the definition of evaluation, 
the evaluator %rho participates In 'decision making destroys his own objectivity 
and, hence, his ut^lllty. Thus, educational evaluatlon'may be defined as the 
process of delineating, obtaining, and providing Information useful to 
decision makers for Judging decision alternatives. 

This systems perspective Identifies four types of decisions for which 
evaluation research can be conducted: planning, structuring, ii^lementlng, * 
and recycling. / 

Planning decisions specify major changes that are needed In a program-^ 
either enda or meana or j^pth. Structuring decisions specify the means to 
achieve the enda established as a result of planning decisions.^ Implementing 
' decisions are those Involved In carrying through the action plan. Bicycling 
dedslom^ are concerned with attalnmenta at any point In a progrsm. They 
are used to determine the relationship of attainment to objectives and 
whether to continue, tepilaate, evolve, ot drastically modify an activity. 




Corresponding to these foidr declslmi types are four types of evaluation: 
*^iput, procMs, and p/oduct. Although specific evaluation 'designs f 
vary according to^fhA setting for change, each has a general purpos 



Kiuct. AXtbougn specixic evaluation 'designs for 
setting for change, each has a general purpose 
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Context evaluation supplies the background for the determination 
of objective^. It provides an Initial basis for defining objectives 
operationally, Identifying program strategies, and developing proposals . 
for outside funding. Context evaluation monitors the, system to maintain . 
a current baseline ot information about it and compares actual and 
Intended system performance. PERT. is one example of a systems analysis ^ 
technique. 

• Input' evaluation prbvldes information for determining how to utilize 
resources to meet program goals. This is accomplished by identifying and 
assessing: (1) relevant capabilities of the organization, (2) strategies 
for achieving program goals, and (3) designs for implementing a selected 
Sitrategy. * One technique useful for input evaluation is co*|^enefit 
analysist 

Process evaluation provides periodic feedback to those responsible 
for Implementing plans and procedures. Its main objectives are to 
. detect or predict effects in procedural design or Implementation, provide 
information foi^ program decision^, and maintain a record of the procedure 
as it occurs. 

' Program strategy evaluation is assessment of the relative efffective- 
ness of different techniques used in a national program. Its goal is to 
Inform those responsible for programs of the different strategies or methods 
used by projects in the national program. * 

Project evaluation is assessment of the effectiveness of an Individual 
project In achieving its Stated objectives. Its goal is to determine 
ending results. ■? ' 

' Project rating is assessment of the relative effectiveness of 
difference in local projects in achieving program objectives. Projects 
operating in similar enviroftaents may be compared. 

The success or failure of a program may be evaluated in terms of five 
categories of criteria: (1) efforts or Input, (2) results of the effort 
or effectiveness, (3) program itopadt in relation to total amount need, 
(4) cost effectiveness or a ratio between effort and laipact, and (5) the 
progrannlng process [12]. ^ 

Evaluation Beaign for Rural Development 

The selection rof an evaluation design and, hence^ its appropriate 
^nd legitimate role cannot be founded on a self-contained, one-time study 
based on"^ before-after comparison of a single stimulus and confined to 
the terminal effects of the program. Nor can it be derived from an 
approach designed . primarily for program' ana^ysM and Pg>gram improvement— 
ilthough these are desirablie activities., ^ /. ^ 3, - * 

To evaluate meatis to assess value. Be£ a j i » ^ , agfelfcment can take 
place, the desired , value must be understood tl3U^lue (^an be assessed 
In several ways duclndinfe ilitultlpn, experiencfe^^^able Jacts,^ and 
research. To ai^s ' ' - «- - 



the yalu.e of a program fi^^^jtf)!^ development on any. 
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basis other than research and the principles of nodem experlaental 
design reduces the probability of distinguishing progriam effects^ if any, ' 
from the effects of other programs and of* the environment. 

In considering the objections to experimenting" design discussed 
previously, Houston [6] makes the. following points: 

1. The criticism that experimental design precludes change ^d , 
Improvement of the program makes the dubious a88U^>tion that modifications 
tend to Improve a pr(|gram, and appears to cdnf use internal with external 
validity. . ^ 

2. That experimental design cannot provide information when it is 
^needed assumes that measurement uid analysis most amlt the terminal ^ 

point of the program. « Subsets of saiiq)llng units provide an acb'eptable 
measure for short-term feedback. ^ ' , ^ : : , 

I 3. Randomization is ofl^en precluded for moral, social, or psycbolo^ 
reasons. Randomization does not require a "no treatment" gtoup — it does ^ 
require that the control group not receive a treatment whose outcomes ^ 
are likely to resemble' those of the program being evaluated. In addi|;iogn^ 
several techniques are available by whicti control-experimental groups <can 
be analyzed [2]. \^ - 

The design cannot be developed » however, until those responsible for 
the program id^tify the values to be assessed in the fashion of ^specif l^p 
goals or objectives, precise fdrmulation of activities capable of achieving 
the specified goals axid the creation of standa];ds against which the program 
can be coiq)ared. Alkln [1], suggests that the evaltiator can help during 
the initial stages by pointing out inconsistencies, potential difficulties, 
or additional data that* mi^t modify the de<^ion maker's views on the 
relevance of certaim^ concerns. Never theless^, the decision maker, and not 
the evaluator, determines the nature of the domain to be examined » 

Administrators have particular heed^and eva^i^tors have specific 
requirements. These needs and requirements can be n|ither in conflict nor 
ignored in the design of the evaluation. Perhaps the role the Rural 
Development Center sh9uld be to provide leadership in the meshing of these 
needs and requirements so that an evaluation design can be 4eveloped that ^ 
provides Information to serve as a rational basis for skiing objective 
judgments on the economic and social costs and effects of a netional 
program or local project for rural development. 

'y 
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< , CHiyPTER III 

-. - „ , , • . • 

ALTERilLTIVE MODES OF EVALUATION AND THEIR APPLICATION 

TO RUiAL DEVELOPMENT* ^ 

Evaluation research^ as an emerging discipline, has not iiad much 
tla6 to solidify its theory, methods, and techniques. Legislative 
pressures have been placed on social action programs [7]. As part of 
the "bowatown" ataosphere surrounding evaluative research, there has 
b«k a great influx of ideas as to iiow evaluation Is to be accomplished. 
These process models are from each of the many disciplines which lay 
claim to evaluative research.. What seems to have been needed for a^ 
time now Is a synthesis and comparison of the various types of models 
used in conducting evaluations*. This paper is a direct attempt to cut 
through the jargon of the multifaceted field of evaluative. research. 

Introduction to Evaluation and Rural De velopjnent 
A .Definition of Evaluation 

For ti» putposes of this chapter, evaluation may be defined as the 
syst^tic examination of a program in operation for thw^purpose of 
Improvement. This process takes the form of asdessing tK^progfap In 
terms of its stated goals and objectives. Such evaluative information 
is then used In order to make decisions concerning program^ improvement. 
There are four key- elements to this definition of evaluation. 

The first of these key elements is that systematic examination In 
evaluation Implies a process . This means that the collection of steps 
used in the evaluation of a program have a logical sequence. Such a 
step-by-step configuration can Insure a better evaluation accomplishment. 

The next key element which needs to be stressed is the concept of 
a proRram In operationt Although this particular idea may reflect the 
authors' biases toward ongoing or built-in program eva«4tion, it is 
believed that this constitutes a critical element for evaluative 
research. When evaluation is an integral part of any program, there 
is a greater probability of examining, ^he program's stated goals and 
objectives than when the evaluation is done In post hoc sense. 



*This article by G. Richard Wetherill, Sociologlst,||^*S. 
Forest Service, Southern Forest Experiment Station, and Joan L. Buttram, 
Evaluator, Training Institute, Narcotic andLDrug Research,' Inc. , is 
reprinted from Rural Sociology in the South; 1976 , Virginia P. Steelman 
(ed.). Baton Rouge, Louisiana: Louisiana State University, 1976. 
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« A third key element to this definition of evaluation is the direct 
orrientatioi^r to decislon^i^king mechanins. Evaluation data, should be 
used in order to faci^^ate decision making for prolan is^tovement. 
However, this partijjdlar definition most definitely leav^ open the 
option for the e.^5|aliator to participate lii program improvement decisions 
or recommendations. Nevertheless, progi^amming decisions must be made 
as befitting t^e role of evaluative research* , 

A fourth and final element o^ this definition of evaluation is 
that of program improvement* As alluded to above, program improvement 
through decisions is an integral part of evaluative research. It is 
the special province of evaluative research to yield objective informa-^ 
tion for program modification decisicrns. There does, however, seem to 
be a dichotomy between evaluation for improvement and evaluation for 
Justification. This issue will be dealt with at another point in this 
chapter. 



A Non-Definition of -Rural Development 

There, is a definite problem having to do with the defining of 
rural^evelopment. While many people seem to know what it is, the 
problem arises lAen they try to set it down in definition form. For 
the Purposes of this, chapter , it would then seem logical to assume a 
common knowledge" definition of rur^l development. Hence, no formal 
defining ^f rural development will be set down in this chapter. However, 
there are certain characteristics of "rural development" which could be 
considered in any such definition. By way of trying to draw some 
bouildaries around' the concept of rural devrelopment, the following 
continuums are offered: from facilities and services-buildipg to 
people-building; ^rom economic factors tp social factors; from govern- 
mentally imposed, programs to Xecal self-help projects, and the list 
could go oti. Basically, rural development is predicated ia a recognized 
Judgment that there are differences and/or disparities between rural 
and urbta sectors. Rural development encompasses programs which are 
directed at reducing such disparities^through planned change. 

' Evaluation Models 

Evaluation, like aaf other developing discipline has adopted a 
particular jargon. Because evaluation is such a relatively nev 
discipline, nuaerous "aodels" of evaluation eriat. Each-»odel is 
adviTcated by a particular grou^ and such aodels tepresent the particular 
perspectives and biases of that group. The tera "«6del" is soiie«hat of 
a BisnoMr. Steele [17] uses the tecps "Wiels," "fraiMworks " and 
approaches to evaluation" Interchangeably. A more accurate descriptor 
than evaluation aodels might be "evaluation processes." Each "aodel" 
is really a description of a process whereby a prograa Is evaluated. 
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TAbles i and 2 provide a comparison a£ aone 21 ■odels of evaluation. 
Although not exhaustive, the models selec^fS for inclusion are representa- 
tive of the" range «f possible model alternatives: Included in the 
description of the evaluation process is a definition or perspective of 
evaluatibh, along with a set 6f dimensions or steps to use in examining the 
performance of a program. In Table 1, information is provided about the 
perspective of evaluation adopted by each model. In Table 2, each model 
is compared to a' sequence of base phases. A brief description of these 
base phases is provided later in this chapter. Although this comparison . 
is an approximation at best, it is hoped that, as such, it will provide 
a useful beginning for a comparative examination of these models. 

While the base phases set up for this chapter axe represented as being 
in linear sequence, the models which are represented in this manner may 
not- necessarily be "fla^." Cycles and dimensionality are represented 
where possible. Overall, however, this representation has as its purpose 
the laying out of - process . 

Base Phase Concept 

In reviewliig the evaluation literature, it was apparent that most 
evaluation models could be best understood through, the utilization of 
a coianon framework or sequence of phases. Each of the models has 
developed a particular jargon and adopted different emphases; l»ow«^epK 
all share certain characteristics. Consequently, rather than to ^^crlbe 
each of the models separately, it was thought that a more useful approach 
for the purposes of this chapter would be to describe some coimnon sequence 
of ftages. By adopting this approach, a more general overview of the 
' process of ei^lpation i& obtained. 

* Phase I — Program Description 

1 

The first phase of the base phase concept in evaluation is program 
description. During this phrase, the'evaluator, with assistance from the 
program staff, describes the program according to its goals and objectives, 
operation, and environment. More simply, the evaluator attempts te-put 
down on paper what the program is all about. This serves^to direc^ the 
evaluator, pro-am staff, and appropriate audiences to the program s 
prlJBry goals and consequent courses of adtion. The result of this 
effort might be considered the program's blueprint. Although, this 
process is seldom carried to such great djitaii:^ Descriptions may be 
graphic; written, or some combination o(f toth. Often this description 
Aikht be the program staffs conception of how the program should exist , ■ 
ideally. It then provides a "standard" by which to evaluate the- program s 
actual performance. Generally, Phase I consists of a written description 
of the program at Soma level of specificity; it then serves as a 
documentat;^ of the goals and operations both to program staff and 
other relevant parties. ^ 
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Tabic 1 

COMPARISOH Of PDIPOSiS OF BVALDATIOll MODELS 



Author 


Purpose 


Alkln [1] 


To provide a reaearcb-orlentedy caae attidy 




approach to evaluation In order to Increase 




the effectiveness of action prograas. 


Allerhand [2] 


To report to progtaa declalon aakefa evidence 




; useful In selecting aaong alternatives ^ 


■Benedict [3] 


- ' ' . ■ 

To establish broad procedures for the conducting 




of the evaluation. « 


Edwards t Gut ten tag , 


♦ 

To provide research-based Inforaatlon to progtaa 


and Snapper [4] 


decision aakirs about possible prograa«lng 




alternatives. 



Falrveather [5] To provide a iietbod of evaluation uhli ^ coMo ares' 

alternative strategies of social Imyyvatlo fc w hit^T 
keeping Individuals at the forefront. 



Gronlund [6] To provide a syst^eaatlc process for the deteral- 

nation of .the ex^t to vfaich already specified 
progrsB goals are achieved. 



Hawmd [8] * To determine ufaethet innovation* Is effective in 

achieving expressed objectives. 



To collect data» sake aeasureaentSt specify 
lapllcatlons and drscv conclusions in relation to 
establlahed progtrsa objectJjgBs and evaluative 
criteria. - 



Lelnhardt [il] To provide a strategy for internal evaluation 

^ich would include the gathering of data ft<m 
«ulti*-discipllnary sources in a coordinated 
evaluative effort. 

» . ^ ^ ' 'i, 



Hoe [12] , To detemine or maiure the quantity of 

act ivl ties » programs » events » inputs* outputs » 
lapact and, to Judge the raJ/M or worth of vhaC 
has, been dime or what teas^esulted. 



It should be noted diat in t}ie preparation of Tables i and 2, the 
Inforaatlon presented is the interpretation of this paper* a two authors. 

^ 32 / 



Bendersbn 
«nd Bond [9] 



2d' 

Table 1 
(contlmied) 



Author 


Purpose ^ 


Provus [131 


^ To determine perf omance of program relative to 
standard in order to sake decisions on whether 
to improve, ma^tain^ or terminate a program*^ 


Rlecken [14] 


To measure de^ixable and uadesinthle^onsequences 
of^an action intend^ ta.advaz^e some\o^ valued 
by tb^ . author . ( y 


Scrlven [15] 


To establish and Justify merit or worth of program 
according to priorities identified during needs 
aasessment, , 


Stake [16] . 


To describe and Judge educational programs based 
on a formal inquiry process. 







Steele [19] To provide data for decisions about the progrte 

in process or for decisions relating to future 
programming strategies. 



Stuff lebeam [19] 



To provide relevant information to decision makers 
regarding program operation. 



Suchman [20] 



To determine the degree of success or failure 
through the Jud^ient of a program of planned 
change in relation to the desired result. 



Tankersley [21] 



To facilitate program adjustments in order to 
increase present program effectiveness and 
increase the probability of success for future 
progr* 



United Rations [22] 



To determine die extent to which the program's 
objectives are being met and* to make an overall 
assessment of the program's socio-economic 
impact upon rural life. 



Ifeiss [231 



' To provide inforution for effective program 
decision making^ 
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Phase II — criteria Foraulatlgn 



The next phase serves to focus th« evaluatioii o^. ceitai^a#ect8 
of thf pro^I» Obviously, -ost rura^ development prog^ kte so complex 
jL?it is^early Impossible to exa«2e sy.tte«a:^ically^^ery aspecj of 
such programs. Consequently, critical; ^r^ l^'^ program's oP^Jf^^ion 
I^e Identified. These critical ar^ b^o-e th6 focus Of the evaluation. 
Standards or acceptable levels of Ij^fptfrnance. piSt also^ developed for 
each of these criteria. For exaiiple. suppose pne of th^: critical 
evaluative areas for a rural development program concerted the Increased 
use of Extension educational services. The program staff would then set 
for themselves a level of accept«We performance (e.ft^se of services 
would increase 25Z) . This criterion serves as a standaty by which to 
Judge the program's performance. The formulation of evaWion criteria 
serves two purposes: (1) it focusea the evaluation effort > critical 
areas and (2)* it sets standards of acceptable performance so program 
can be evaluated. 



Phase III — Evidence Gatherlrfg 
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In the third phase of evaluatioa, evidence is gathered concerning 
the perfomance or operation of the program according to the e^luation 
criteria fonnulated and es^hllshed in Phase II. More simply. ^ata,are 
collected for each of the specif ied evaluative criteria. Consjru 
of instruments and appropriate statistical analyses ^f colllectAd 
occur during this phase. The evaluation is prohahly more similar Xd 
the traditional research investigation during this phase than during 
any other phase. The skills and knowledge required of the ev*luator 
are the principles and mechanics of research, design, methodology, and 
analyses. 



Phase IV ~. Judgment>Making 

The criteria established in Phase II and the evidence (or data) 
collected during Phase III are used in making decisions regarding the 
performance of the program in Phase IV. Basically, the data (or actual 
performance) of the program are compared to the established criteria 
(or ideal performance) for that program. Obviously, three conditions 
could exist. The program could fall below a mi n inti ta acceptable level of 
performance, attain expected performance, or exceed projected performance 
Decisions regarding the significance or effect of thest performance 
discrepancies would be made. ;The models probably differ more during this 
phase than in any other phase. Some models p/ovide for the evaluator 
simply feeding such information back to the program staff. Others 
encourage the evaluator to go beyond this feedback role and assist in 
the decision-making process itself. In all cases some determination 
of the program's merit is made during this phase. 



ERIC 



36 



30 



Phase V - Dissemination 



The last phase of evaluation Is the dissemination of findings. 
Although very few models explicitly lay down guidelines or procedures 
for the reporting of evaluation results. It was apparent that' this must 
always occur. Findings are usually reported back to clients* Special 
contractxjal arrangements are sometlnies made to Include wider audiences* 
Trie use of the evaluative findings depends upon the status of the 
program operation. If the program Is terminating^ the evaluation Is 
viewed as summatlve and Is a documentation of final assessment of the 
l>rogram*s performance. This sunmatlve Information might be used by 
other similar projects In designing and Inqplementlng their own particular 
"programs. If the program Is continuing, the evaluatloir-J.s viewed as 
fqrmatlve and as a source of Information for program jbaptovement. 
impropriate changes can be Implemented' and ^ evaluated and/or other areas 
of program operation and performance can be examined. 



Implications for Rural ^development 

The i]q>llcations for program evaluation in rural development are 
numerous. Because the major ^cus of mast evaluation models •is to 
document systematically the operation of the program according to some 
established criteria, decisioxis regarding the future operation of the 
program ate facilitated by evaluation efforts. Beioje decisions can 
be made regarciing program changes, doctoientatlon concerning current 
program activities must be obtained. The reason for such documentation 
is to provide evidence regarding the faithful Installation of the 
program. If in fact all of the necessary components were not present 
an4, the program consequently did not achieve its go^ls, those components 
Initially bypassed can be Implemented and the program re-evaluated* On 
the other hand, if all of the necessary components were present and the 
program still did not achieve its goals, then systeutic changes ean be 
initiated which would allow for program improvement. Evaluatiot^ would 
then be ongoing until the stated program goals were achieved. As a 
third possibility. If all of the necessary components were present and 
the program achl^ed its goals, evaluation becomes a monitoring process 
to Insure continued effective operation 1 

» 

Anot^her view of evaluation useful to rural development is that of 
evaluation for the purpose of contlmiedmprogram support. Although 
realistically evaluation for^such a purpose is valid, the primary foctis 
of this paper is that of defining elraluation as Integral to program - 
lq>rovement. It must be remembered, however, that program Improvement 
and program justification are not mutually exclusive concepts. The same 
evidence which was gathered for program Improvement can also be used for 
justifying continued program support. 




A generalized example of the aho^ fqj rural development might 
fall into the realm of the pilot developmeat project. In many cases, 
the continuation of such pilot prdjects on h larger scale is dependent 
upon the demonstration of program effectiveness. It is the nature of 
such a pilot project to act as a proving ground for rural development 
strategies. Based on the evaluative Information obtained from the 
pilot project, only"thoa0 strategies lifhich have proven themselveQ 
effective at the pilot stage are tested on a larger scale. Cpnpequently 
greater probabilities of success are guaranteed. This is especially 
import^t when limited resources are available. 

- V . ■ - . • : 

Conclusions • 

While this paper has not attempted to take an advocacy position 
either for the base phase concept of evaluation or for any of the 
individual models presented herein, it is hoped that the similarities 
among models was apparent,. If it can be possible for those working in 
the ^ireas of rural development and evaluation to cut through the jargon 
of the different orientations %o evaluation, then meaningful advances 
can be made for development. Evaluation is then seem as a useful tool 
for rural development. Hot only does it provide for the systematic 
evaluation of particular rural developfoent programs, but it) also allows 
for the selection of more effective rural development strategies. 



38 



32 

SEFEREtlCES 



1. AlfcliH Marvin C. 

1969 /^Evaluation thaory davalopnent.** Evaluation C o M cn t 2 

(October):. ^2*7., 

2. Allerhand^ Melvin S. ) 

1971 "The proceaa outcoaa reaaarch aodel: an altarMtiva in 

evaluative reaearch (a caae atudy).** Pp. 131-147 in Richard 
0*Toole (ed.). The .Organisation, Manageaent, and Tactfca of 
i Social Reaearch. Caabridge, Haas.: Schenkaan. ^ . 

3. Benedict, Larry 6. 

1973 ^**The Fortune/Hutchinson evaluation methodology: a decision 
oriented approach.** Paper preaented at a Syapoaium of the 
Aaerican Educational Reaearch Association, Hev Orleana. 

4. Edwarda, Ward, Harcia Cuttentag, and Kurt Snapper 

1975 "A decision-theoretic approach to evaluation research.*- 
^ Pp. 139-182 in Elaer Struening and Marcia Guttentag (ed8.}| 

Handbook of Evaluation Reaearch, Vol. 1, Beverly Hills, 
California: Sage. 

5. Fairweather, George W. 

1967 Methoda for Ezperlaental Social Innovation. New York: 
John Wiley and Sons. 



JS. Gronlund, Horaan E. • ^ ^ 

1965 Meaaureaeiijt: and Evaluation in Teaching. Nev^qrk: MacKLllair. 

i A 

) 

7. Ha—in, Anne E. 

1975 **Legialation specif iea prograa review and evaluation.** 
■eMllne 3 (January): 1-10. 

8. Ba«0nd, lobert L; 

1971 ''Evaluation at the local level.** Aicaon, Arizona: Epic 
Evaluation Center (alMeograpfied) . 

9. Endaraon, H. A. and B. J. Bond' 

1966 **Evalttation developaent prograaa.** Journal of Cooperative 
Eztenaion 4 CKlnter) : 233-239. 

10. Ettox, Alan B. * 

1969 **Contlnuotta prograa evaluation.** Pp. 368-391 in Nathan C. > 
Shaw (ed.), Adainiatration of Continuing Education. 
Hlahlngton, D.C. Rational Aaaociation for Public School 
> Adult Education. * ' . ■ ^ 

11. Leinhardt, Gaaa ^ 

1975 **A Strategy f or Prograa Evaluation.** Paper preaented at, 

the annual Mating of the Aaerican Sociological Aaaociation, 
San Pranolico. «. 



ERIC ' '39 



33 - • 

12. Moe, Edward 0. • , „ > ^ ^ 
' 1974 "The Evaluation of Developiaent ProgramB."' Tiapcr presented 

at the Workshop on Evaluating State Title V Pilot Programs 

in the -NoiftheaBt, Ithaca, Hew York. 

13. Provus, Malcolm ' ^ 

1971 Discrepancy Evaluation for-'Wlucational Program Improvement 
and Assessment. Berkeley, CaTifomia: McCutchan. ^ . /f?n/ 

14. Riecken, Bi^nry W* 

1972 "Memorandum on program evaluation." Pp. 85-104 in Carol H..-. 
Weiss (ed.). Evaluating Action Programs: Readings in Social 
Action and Education. Boston: Allyn and Baco»« 

15. Scriven, Michael ^ 

1967 "The methodology of evaluation," Pp. 39r83 in R. W. Tyler, 
R. M. Gagne, and M. Scriven (ed.). Perspectives of 
Curriculum Evaluation* Chicago: Rand McHally. 

16. Stake, Robert E. 

1967 "The countenance of educational evaluation." Teachers 
College Record 68 (April) : 523-540. 

17. Steele, Sara 

1973 Comtemporary Approaches to Program Evaluation: Xmplications 
for Evaluating Programs for Disadvantaged Adults. 
Washington^ D.C.: Capitol Pxiblications. 

18. 1970 "Program evaluation - a broader definition." 'Journal of 

Extension 8 (Summer): 5^17. 

19. Stufflebeam, Daniel L. 

1970 "Programmatic change." Paper presented at the annual 

meeting of the American Vocational Association, New' Orleans. 

20. Suchman, Edward A. * 

1967 Evaluative Research. Hew York: Russe|ll Sage Foundation. ^ • 

21. Tankersleyt^ Howard"^. 

1974 "Evaluation Needs Under Title V of the Rural Development ^ 
Act of 1972." Paper presented at the Workshop on Evaluating 
State Title V Pilot Programs in thej Northeast , Ithaca, NY. 

22. ' United Nations ^ \ 

1962 ^Report of the Asian Seminar on Planning and Administration ' 
of National CoBBitnlty Development Programs: Bangkok, 

Thailand, 1961. New York: United Nations. 

/- - ^ - 

23. Weiss, Carol H. 

1972 "Evaluating educational and social action programs: A 
treeful of owls." Pp. 3-27 in Carol H. Weiss (ed.), 
Evalixatlng Action Programs: Readings in Social Action and 
Education. Boston: Allyn and Bacon. ^ 



ERLC 



40 



I CBAPTER I 



L2r 



RESPONSIBILITIES OF THE EVALtUkTIOM IffiSEARCBER: 
THE A.S.A. COQE OF ETHICS AS A GUtSELINE* 



Many Aaericaas, vere shocked dally as the tragic story of Watergate 
unfolded. Aaerican society is still going through the process of 
assessing the raalficatlons of Watergate. Perhaps In this aftemath 
of public distrust of govemaentt ve should exaaln^ practices of sociologists 
that could be called. Into question on both aoral and ethical grounds, if 
not legkl grounds* 

Sone ethical issues such as the peer-reviev system for grants, use 
and alsuse of science by govemaent, and the protection of human subjects 
are already being confronted In academic and non-academic circles. .The 
peer-review system by vfalcb federal agencies dispense funds to outside 
scientists has come under attackr by Congress. Senator William Prozmlre 
has charged that the peer-revl^ panels us^d b/ the National Science 
Fotindatlon are packed with r^pres6itatiyes from those universities that 
obtain the grants. At the present time the National Science Foundation 
is conducting Its own Internal review of .this situation [7]. 

Two recent books. The Brain Bank of America , by Phillip M. Boffey 
is] and Advise and Dissent by Joel Prlmack ai^l Frank von Hippie [20], 
focus upon significant dimensions of the. relationship between govemaieut 
and science 9 as well as the use and misuse of science. Boffey *s book, 
which explores the role^ thi^ National Academy of Sciences, points out 
that the academy allows federal agencies to co m m en t on early drafts of 
reports and In the past has even permitted ag^icies to suppress reports 
they did not like. Boffey would like "the Acad^ to bring the nation's 
best scientific talent to bear on societal problems and then enunciate, 
unflinchingly and unequivocally, the nearest possible approximation to 
the truth" 112]. 

Prlmack and von Hippie content that federal agenci^^. do not really ^ 
want advice, but Instead wmit prestigious^ groups to legitimise decisions 
that have already been made. Wh«a "Independent*' advice is given i^iay 
be distorted to suit the purposes of the decision, maker. Prlmack axii^^ 
von Hippie further contend that the process of advising must be made \ 
m6re public and open to a broader set df viewpoints. Lobbying before/ 
Congress is seen as necessairy to -keep executive agencies and their- 
captive scientists honest. / The advisory processes need to be^re open, 
'participatory, and responsive to the public interest, according to Prlmack 
and von Hippie {12]. 

Anbt^r Issue that trsfiscoids disciplinary boundaries concerns the 
protection, of the rl^ts of researdi subject. Donald Warwick [25] contends 
that deception is common In social science research and that it not only 
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Bural Sociology Section, Southern Association of Agricultural Scientists, 
February 1-%,' 1976 at Mobile, Alabama. 



takes place in the laboratory button the street and In our social 
instltuttons* Warwick challenges the idea that the end Justifies 
the means. Studies such a^ Fytealjon in the CUssroog and Tearoom ^ 
Trade have been called into quest ioft^on ethical grounds. In 1973 f 
the American Psychological Association outlined new ethical respons- 
ibilities for researchers using human subjects. It was recommended^ 
that deceptive practices usc^d to engage subjects in research be elim- 
inated. Warwick ends his essay by noting: % 

At present we too often dispose pf ethical questions 
quickly so .that we can get on with the real business of 
theory and research. The time has come to examine hot 
only the techniques, but theitooral l]q>licatioii8 of social 
research. Watergate was the latea^xample of corrosive 
deceit in America. In the social ^^Bnces, as in politics,^ 
the truth is often sacrificed on tM|6ltar of some higher 
principle. Tte cumulative results ate a pervasive suspicion 
of government and an increasing wariness in dealing with 
our fellow man*. These are the natural fruits of a deceiving 
society. Social scientists who? do not hesitate ts-^K)int an 
accusing finger at the White House are too quick to shrug 
off their own coBq>licity in this moral decay. We should not 
put our own house in order with a permanent moratorium on 
deceptive research. _ 

fiiven that questions of ethics are confronting researchers every day, 
the purpose of this paper is to review the code of ethics of the American 
Sociological Association and to explore areas of responsibility 
dealing) w^th evaluation researchers. 



Trends in Employment 

In the future more and more sociologists will become involved in 
evaluative .research. This prediction is based on two rather cl^r trends. 
First, the unemployment! of Ph.D. sociologists in the academic^ world is 
on theAcrease [IQ, 11, 13, lA, 16]. "Unless demand booms or the'birth 
rate deceases sharply in the immediate 'future, by 1980 Jthere will.^e 
approximately 1,200 to 2,000 sociologists who have earned the Ph.t>., 
degree, but .who catmot find a career in sociology," [13]. Panian and 
DeFloer [19] note that: > ^ 

It ^ \ 

given a continuation of cturrent* political conditions there^ 
is little doubt that we are \f£clng declining enrollmerii^s in 
colleges. It is not likely that colleges and universities . 
will receive subsidies in the same amount as people with 
advance degrees. As tax r^enues and support for private 
institutions continue to be tight, colleges and tmiversitles 
wlll)i,be forced to cut expenses and departments will not 
add new faculty positions. More and more* therefore, 
academic employment £s becoming a Mtter of replacement 
^"^rather than new growth. Even the replacement market may 
eventually decline if present trends continue. • 
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Given the grla acadenlc aarket, tbe sociologist of the future will 
be increasingly forced to turn to non-acadeaic eaploynent. At present, 
roughly 25 percent of the total estlwted nuaber of sociologists have 
taken non^^cadealc ei^loyaent and this^ proportion is lllcely to increase 
[19]. '*The majority of the non-acadealc sociologists vork for non-profit 
(tion^religious) organizations or govemaent (federal* state and local) 
agencies in a research capacity [19] • 

Second, the availability of funds to do evaluation research and the 
deaand for program evaluation vilL create an avenue of eaployaent [S]* 
As Heison Foote suggests '^froa projecting recent eiq>eriences, the principal 
non^acadeaic sodrce of eaployaent for Sociologists will be in perf oraiAg 
evaluation studies,, primarily for federal agencies. The recent abundance 
of such studies reflects the abundance of federal programs and sub- ^ 
prograas, purportedly experimental in nature, vfaich seea to require 
evaluation in order to deteraine if they should be '^ctmtinued, expanded, 
contracted* or teralnated**. Walter Wllliaas in his book. Social 
Policy Research and Anaylsis [30 ]» has already raised the question as 
to **wWAier the various teaching instltutid^s in the social sciences 
researdDi cdaauhity vlll actAto-incirease slgpif icantly tbe,supply>a^policy- 
oriented researchers." 

In that aany of the sociologists in graduate school today vUl be 
employed in a non-acadeaic setting doing efshxatife Research periiaps it 
would do the profession well if aore colleges and universities bega^ 
offering courses in evaluation research and as a part of such a cdurse, 
address soae of the ethical issues that will confront the sociologist 
qua evaluator. 

Those that have already eabarked on careers In evaluation research 
realise that ^be field is besieged with a host of probleas; conceptual 
and aethodological probleas, problSas of relationship, status, and ftmction, 
practical, probleas* probleas of career and rewards [26« 27» 28]* To ' 
speak of special ethical considerations only adds to the list. Evaluation 
reports are becoalng front page news. Sociologists whose evaluative 
studies enter the political arena must l>e prepared f oi: search£iig scrutiny 
of their aethods and techniques [26 ]• 

Saad Hagi and Ronald Corwin, in their book* The Social Contexts of 
Research [17]» atteapt to analyse the way in which soical contexts influence 
the research process. They note that there exists a paucity of material 
tQ acquaint future researchers with their anticipated roles and the - 
constraints placed upon them as researchers. J V 



Code of Ethics 

Tbe Code of Ethics adopted by tbe ASA In 1970 prov^|^ some guidelines 
for future researchers. The preamble of tlje code [21 sudclneay argues 
the nemd for a code; 

Knowledge is a form of power, and in a society^ increasingly 
dependent on knowledge, the control of information cteates the - 
potential for political Mnipulation. For thes^ reasons, we . ^ 
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affirm the ttutom>iny of sociological inquiry. The sociologist 
must be responsive, first and foremost, to the truth ^ his 
investigation. , Sociology must not be an instrtxaient of 
person or gfowp which seeks *t<f suppress or misuse knowle4ge. 
Thi fate of sociology as a science is dependent upon the f/ite of ^ 
" free *inquiiry in an open society (316) . ^ 

Sociologists should become faMliar with the code beqause it does 
provide some guidelines. The "code" [2] states: 

• , Code of Ethics 

1. Objectivity in Research ^ 

4 In his research the sc^iologist muiv maintain scientific objectivity* 

2. Integrity in Research* 

The sociologist should recognize his own limitations and, when appro- 
priate, seek more expert assistance or decline to undertake research 
beyond his competence. He must not misrepresent his own abilities or 
the compet^^ce of his :«taff to conduct a particular research project. 

3. Respect of the Reseairch Subject ^s Right to Privacy and fiigriityy . 
Every person is entitled to th^^righi of privacy and dignity of 
treatment. The sociologist must ifespect these rights. 

4. Protection of Subjects from Personal Harm ' • 
All research shquld nevoid causing personal harm to stibjects used 
in research.^' * ^ 

5. Preservation ol Confidentiality of yesearch D^ta 

Confidential information provided J)jt a* research subject must be treated 
as such by the sociologist. Evemthough research information is not a 
privilege communication under jtfa^JU the sociologist must, as far as 
possible, protect sub^ects aiuf informants. Any promises made to such 
persons imist be honbre^; hbwever, provided that he respects the assurances 
to witldiold information^ pf misconduct o|^ individuals or organizations. 
If infora&nt or other subject should wish, hoiimver, he can formally 
release the researcher of a promise to all mdfeers of research organizations 
(i.e., interviewera, coders, ^erical staff, etc.^, and it is the responsibility 
of the chief inyestigators to B|e that they are instructed in the necessity 
and imp#i:tance c^f .maintaining confidentiality of data. The obligation 
of the sociologist l^t^iludes the use and storage of original data to 
^^^^ which a subject *s name is attached. When requested, the identity of an 
. organization or subject must be adequately disguised in publication^ 

6. Presentation "of Research Findings ' - . n 
The ^Qciologist must present his findings, honestly and without dis- 

^ tortion. There should ue no omission of data from a research report ^ * 
which might significantly modify the interpretation of findings. 

* • * * 

7 . Misuse of Research Role ' , . 

■ ■ * ■ f i > ■ « 

The sociologist nust not use his role as a cover to obtain Infomatlon for 
other than professional purposes. 
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8. Acknoirledgiient of Jteaearch Collaboratloa and Asslstaace 

The sociologist wist acknowledge the professional contributions or 

assistance of all persons vho collaborated in the research. 

9* Disdosnre of the Sources of JPlnanclal Support 
The sociologist aust report fully all sources of financial support in 
his res^rch publications and any special relations to the sponsor 
that affect the Interpretatim of the findings, 

10." Bistortion of Findings by Sponsor 

The sociologist is obliged to clarUy publicly any distortion by a 
sponsor or client of the findings of a research project in which he h^s 
^participated* 

. 11* Disassociation from Unethical Research Arrangeaents 
The sociologist oust not accept such grants , contracts, or research 
^assignaefats as appear likely to require violation of the principles 
above, and oust publicly terainate the work or foraally disassociate 
hlaself f roa the research if Jbe discovers stich a violation and is 
unable to achieve its correct^!bn. 

tl2. Interprj&tation of Ethical Principles 

When the aeanlng and application of these principles are unclear, 

the sociologist should seek the Judgaent ,of the relevant agency 

or coaaittee designated by the American Sociological Association. Such 

consul tatiorn, iiowever,' does not free the sociologist froa his individual 

responsibility for decisions or froa this accountability to the profession. 

13. Applicability of Principles 

In the conduct of research the principles annunciated above should 
apply to research in any area eitlver within or outside the United 
States of Aaerlca (318).' 

In reference to the "code," Jane Record {21] contends that professional 
societies recoil from the enforceaent of codes. The enforcement of 
t^ code has been a problea over the years' [3, 4, 6, 22]* The Association 
passed the "£ode" but did not create enforceaent policies. According to 
Record, "in the filial analysis the Integrity of scholarly inquiry can 
be protected only /by the courage of morally accountable individuals.** 

S A 

Special Responsibilities of Evaluation Researchers ' 

Those soclaloglsts engaged in evaluation research whose work falls 
Into the public area aust be such aorally accountable Individuals. They 
»ist no.t duly be aware of the before-mentioned prob^eas but of ^he speclaL 
Interest groups and their relationships with acadealc resesitch Institutions 
that ml^t affect thelx. work. How of tea special Interest groups '*have 
attempted tp frustrate inquiry, to dictate its shape, to suppress or modify 
its flndlngf, and with degree of success. Is difficult to deteralne'V [21]. 
Evaluation Researchers have to 1>e aware of the political context of such 
research > tM reasons ai to why a prograa is being evaluated, and to what use 
the evaluation' #111 be ,put% 
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The no8t that a code of ethics can do Is to highlight respoiisl- 
blllties. Evaluation r6seai;chers' responsibilities go beyond any cc 
because of the special nature of their, research. There exist many ateas 
where professional responsibilities beyond the code are subject to . 
question. Five such areas are discussed below. \ ^ 

First, evaluation reports should be understm^^bj^ Those that 
engage in evaluation should cowmlcate the results of their work so 
that It does not have to be translated before It can be used for policy 
decisions. Etzlonl [9] has complained that the «ore analytical sciences 
use to esoteric language ^hat mist be translated before It has any 
meaning for policy makers. 

Second, evaluation, reports should be completed on time so that 
policy makers can reach early decisions. "For thsr^sic researcher, 
science is an open-ended enterprise. There are no uitrlnslc reasons 
for the coq>letlon of a study at any particular deadline. For the policy 
maker there are specific times -when fimdf^aental decisions will be made 
and the decisions made then will become the base for more detailed 
decisions. The policy researcher »ist schedule his research so as to 
produce conclusions by that point" [9]. 

/ 

Thirds effective methods for presentation of findings and dlssiml- 
nation of findings need to be developed . Researchers should file a 
report not only with the, program under study or the gr^tlisg <9g[encles 
but with similar programs as well. Too many reports end up on shelves, 
never to be read or utilized. If a researcher suspects tMt a report 
wllj be suppressed, those avenues shoiild be explored that will Inform 
tkose people in power about the contents of the report [26 1 • Ldngood 
and Slatael urge the evaluator to become an advocate for his results 
and to take part in the rough-and-tuiid>le world of organizational 
decision making flSJ. 

Fourth, avoid entering into agreements with organizations that have 
the rights via agreement under contract to censor your report . The ASA 
committee on ethics has stated that "research administrators should not 
censor or suppress monographa and research papers produced by the 
sociological investlgatl6n in his program or agency except in the 
interest of scientific quality and objectivity, nor sho^d the profes- 
sional investigatoir tolerate such censorship" [21]. Harold Orlans [18], 
In an article dealing with ethical problems of research sponsors and 
researchers, concluded that not "all virtue lies with the pursuit of 
academic knowledge," not all vice with the use of knowledge for praptlcal 
purposes, and that the main problem demanding attention is npt how to 
get social scientists more money with fewer restraints. The sponsors 
of research are not only as hiunan but as moral as we who ask for money: 
"their, motive and Ineradicable offense is not their wiclsdness but their 
power." » 
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Fifth, researchers need to learn to work with prograa adalnlstrators 
In a oeif relationship so that research reports will be aore aeanjLnRful 
for policy analysTs^ VlUlaas [30] contends that research contribution 
Is only possible If social service researchers and the federal bureaucracy 
can work In a ^er relationship. He suggests that In «i agency soae 
aechanlsB (a staff office) Aist be established In order to create a ^ 
good working relationship between the social science researcher and th^ 
federal bureaucracy. '*Pollcy analysis Is a scans of synthesizing Inf^- 
nation l^jj^uding research results to jnroduce a foraat ffiv policy 
declslonrtthe laying out of alternative choices) and of determining 
future needs for policy-relevant Inforaatlon" [30]. Unless the two entitles 
learn to^work together many research reports will have only one perspective 
and that perspective perhaps vlU not be meaningful for determining 
policy choices. ^ 

Susaary \^ 

In this polit-Vatergate era ethical issyes coitfrcmt every reisearcher 
In a variety of different vays. Sociologists in the future, because of 
lack of eaployaent In academic setting, vUl perhaps be conducting sore 
evaluative type research. Sociologist In gra^luat^ school today should 
become familiar with .the **Code of Ethlcs*1 of tBi^^A and Investigate, 
eie|llore, and discuss other Issues (such as the^^es mentioned above) that 
will confront thern*^ — The reality today and perhaps In the future Is 
well put by Becker. Ethical cannons **waBt remain a matter of Indlvldiial * s 
judgment" [23]. Let's hope that individuals, making such judgments at least 
vlll have a reference point. 
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CHAPTER V 



SOURCES OF INFORH&TIOM FOR EVALOATISG RURAL DEVELpPMENT: 

AN OVERVIEW* 



This handbook of evaluative researcli on ruiial development sources 
of inf onaanrion is intended as an evaluator guide to nater lal vhteh 
we have found tp be nost helpful for locating. evaluative research < 
reports » articles and reference booksiNEaeh 6f the sourcea Included 
has been utilized in the aaaaalng of evaluative research referoites 
in an ongoing Southern Rural I>evelopBent Center p^TiOjy^'j **A Synthesis 
of Evaluative Research Literature for Rural Developn^it in the Southern 
Region." • ^ \ 

Included in this handbook is detailed inf onation a^^t co^)uterized 
information retrieval systeas vfaos^ std>ject Batter is relevant to 
evaluative research;-^^ listing of those sources of library' Infom^tion 
whiqh are - axxA which are not - useful for evaluatlive researchers; and a 
br^:fef selected bibliography of references for further information. 

One of the major objectives of the present project is to increase 
awareness of the methods and types of evaluative research literature 
available to program developers and evaluators. We hope this information 
guide will help evaluators and other social scientists use their 
bibliograi>hic search time more efficiently by pointing out sources which 
haye the greatest potential for output. 

Computerialed Information Retrieval Systems 

Included in this secticm are detailed descriptions of four 
lUformatioiQ-shaTing systems which contain many sources related to the 
evaluation of rural development. Like most information retrieval 
systeas » these four deal with rather specialized areas of itudy. For 
infomatlon concerning the hundreds of other computerized information 
retrieval systeas of fering, searches In nearly aa ma^y subject areas, 
the researcher should see: ^ j 

Kruzas, A. J. (ed.) 
1974 The Encyclopedia of Information Systems and {services . 
Ann Arbor, Michigan: Anthony J. Kruzas Ass6ciateb. 



♦Paper presented by Laura Graybum, Undergraduate 
Marilyn Magee, Research Assistant, and Myma S. Hoskins, ^ 
Assistant, all of the Department of Rutal Sociology, Texas 
University at the Annual Meeting of the Rural Sociology Sec 
Southern Association of Agricultural Scientists, February 1 
at Mobile, Alabama. 
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A tiord might be offered at this point concerning which "keywordar" 
and ••descriptors" tap the •ost relevant evaluative literature indexed 
In these information system. Listed below are these Tteywords : 
BVALOmOH. RURAL DEVELOPMErt. POLICY. PROCTAM. ACTIOH RESEARCH. PROGRESS 



S mithsonian Science Information Exchange (SSIE) 

Director: ^ David P. Her*«!rr-Bi.D. . President 



Address: Smithsonian Science Information Exchange 

1730 M. Street. Horth West. Room 300 . 
•f., Washington. D.C. 20036 

Telephone: " (202) 381-5511 ^ 

Subject Matter : Ongoing research, sponsored by both f^eral and 
non-federal organisations, in all fields. SSIE answers questions about^ 
who is working on what problems, at what Ideation, and wi^what support 
It organizes, tabulates, and suMsrizes research support aiwK^repares 
surveys of research in broad areas. SSIE prepares material by\contract 
for publication in response to requests for catologs of ongoing research 
W various organizations. It carries on a proAam of issuing periodic 
^formation in the form of selective <^lasemli)rftidn searches, and it 
maintains a registry of basic and applied research being undertaken. 
SSIE covers life, physical and social sciences, including medicine, 
agriculture, education, community programs, engineering, water resources 
and other traditional science areas. Coverage is limited to projects 
^t an early stage, prior to research results and publications. 



Turnaround Time: About two weeks. 



cost : Cost is $50.00 per search which yields up to fifty notices 
of rei«aich projects. $10.00 for each additional notice of research 
projects. Presearched packages are avalUble (not cuatomized. 
however) at ^ower costs. . 

Av alUbUity : Services are available to All; however, an entire ^ 
program of a ^^^ting agency may not be released without that agency s 
pttSsalon. SSrch services may be ordered from the SSIl by letter, 
telephone, or by c«He <Telex 89495). 

Utility: Of the Wjwencea retrieved, approxlmtely 631 were 
included aa useful sootcee In the present project's evaluation 
bibllofrapfay. 
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Edwatlonal Resources Inifaraatlon Ceikter (ERIC) 



Director: 
Address : 



Ever^ 1 .^ligton 



Telephone: 



ERIC "Clearlngl^use on Rural Education and Siaall Schools 
Box 3AF 

New Vexifio Stkte University 
Las Crue4«, 1^ Hezico 88003 

(505) 646-262^3 



Subject Matter : The Educat^ional Resources Information Center (ERIC) 

ich covering a specific area. The ' 
on Counseling and Personnel Services; 
ition^ ERIC Clearinghoi:^e on Early 
ighouse on Educational Manageaent; ERIC 
and Gifted Children; ERIC Clearinghouse 
ringhouse on Information Resources; ERIC 
ges; ERIC Clearin^ouse on Languages and 
ise for R ead ing and Co— mlcation Skills; 

Science Education; ERIC Clearinghouse on 
ingbot^e on Tests, Ifeasureaents , and 
LOuse on Rural Educati<m and iSmall Schopls 
most Important for this evaluation 
IC/CRESS acquires y Indexes » and abstracts 



has sixteen different scopes, 
scopes are: ERIC Clearinghous 
ERIC Clearinghouse on Urban Ed 
Childhood Education; ^IC Clea 
Clearinghouse on the Handicap; 
on Higher Education; ERIC CI 
Clearinghouse for Junior Col 
Linquistics; ERIC Clearin_ 
ERIC Clearin^ouse for Socia 
Teacher Education; ERIC CI 
Evaluation; and ERIC Clear 
(CRESS). Of these scopes, 

project is the ERIC/CRESS. «^j.wuiME»i» acsuxj^tso^ 

information related to all aspects of American Indian education^ educa- 
tion in small schools, Mexican ^Amer lean education, migrant education, 
outdoor education, and rural education. Research reports, program 
evaluations, curriculum guidesL instructional materials, professional 
information, and many other klSids of information which would be of 
value to an educator or educational researcher are gathered, evaluated 
and made easily accessible through the ERIC indexes; RIE and CIJE. 

The Research in Education (RIE) is a monthly Journal published by 
the U.S. Government Printing Office, containing resumes and Indexes. 
Each item is Indexed by subject, author, or principle researchars, 
and sponsorteg institution. Complete texts of most documents cited 
in RIE are a^^able from the ERIC Document Reproduction Service (EBRS). 



The Cui^6i5i|43ip^^ to Journals in Education (CIJE) provides 
detailed indexing by subject and author for articles appearing in over 
seven hundred educational and education-related Journals. Users slKwild 
use their local refeourqe centers and libraries to locate the Journals 
eit^. The CIJE index ]ta published monthly by Kaxmillan Information 
Rivierside, New Jersey, 

Turnaround Time ; Prom ten days to two weeks. 

Cost: If the conqmter search is used, the cost var 
to state from $15.00 to $30.00. 
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Availability : It Is available tb anyone wishing to use It, In ) 
fact 9 flumy ^resource centers and libraries have the ERIC Indexes 
available and also maintain the cooplete EltIC microfiche collections 
which may be utilized by the public. Users can obtain hairdcopy print- 
outs of Information contained on f iche. at cost, depending on the 
article* 

Utility ; Of the references retrieved^ approximately 35Z were 
included as useful sources in the present project's evaluation 
bibliography. 



Current Research Inf ormatioa System (CRIS) 
Director: Mr. John Meyres 

Address: Current Research Information System 

United States Department of Agriculture 
Washington, D.C. 20250 

Telephone: <202) WI-lllZ 

Subject Matter : CRIS Is a centxalized, computer-based agricul- 
tural research information system administered by the USDA-Cooperatlve 
State Research Service (CSRS) for the benefit of State Agricultural 
Experiment Stations of Forestry, LandsGrant Colleges of 1890 and 
iWkegee liistltute. All 53 State Experiment Stations and the USDA 
furnish forms for all active research frojects. Research projects are 
submitted annually and are entered intp the co^^teirlsed Information 
management, system. For each work unit for which information Is 
requested, the following must be specified: 

1. Name and address of recipient ^ 

2. Date information Is needed 

3^ 4 statement ot informal needs (be as specific as 

possible, within 100 words) 
4. Level of research re<ittired 

Requests are edited by report analysts at CSRS to assur^ that 
proper codes for coModltles, activities, research prc^lem areas, and 
keywords have been used to identify relevant research bein^conducted 
in other states. These facilitate rapid retrieval of the information 
in a form which is useful for the requesting organizations. For each 
request CRIS will give the name of the principal investigator, 
performing institution, and department, and will also give a brief 
description of the project-title, objectives, plan of work, and 
current progress, including the mor/s important recent publications. 

Turnaround Times TWo to four weeks. 
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Cpst and Availability : This Information is available to those 
affllllK:ed with the State Agriculture Experiment Station at no cost 
by submission o£ informal retrieval forma to CSRS«^ 

Utility : Of the references -retrieved, approximately 80Z were 
included as useful sources in the present project's evaluation 
bibliography. 



Cataloging ^xld Indexing System of the Katicmal Agricultural Library (CAIN) 
Director: Ronald J. Walton 

Address: j Cataloging and Indexing System of the National 
^ Agricultural Library 

National Agricultural Library 
* > United States Department of Agrjj^culture 
10301 Baltimore Boulevard 

Beltsville, Maryland 20705 , 

' ' ^ • V 

Telephone: (301) 344-3750 ^ 

\^ 

Subject Matter : Literature search services have been primarily- 
in the form of current awareness profiles. These profiles have been 
made available to U.S. Depaftme^t^f Agriculture researchers through 
the Department's Agriciilti^al Resek^ch Service. General public access 
has beiKi available through the Universities of Florida and Georgia, 
and a^^giQiber of .other loca^ona ai^ound the world. 

On-line interactive bibliographic search and retrieval service 
was put into operation July 1973. This data base consists of 16 
magnetic tapes ^fiilch contain catalog and bibliographic data on all 
monogMphs apd many journal articles received by the National 
Agric^4:ural Library. This file eontains over 700, OQO records. 

The subject category most ^relevant to the evaluation of rural 
I development programs is 0505 General Agriculture and Rural Sociology . 
Including human ecology, social psychology,^ social effects, social 
institutions; rural organizations and movements, leadership, farmer 
political power; health, welfare, etc., affecting family life; ruraj- 
urban relations; population migration to urban areas; social aspects 
of migratory and contract labor, women as* laborers, sharecroppers, etc. 

Cost and Availability ; Free of charge to eogployees of the OSJ)A 
and to on-site users, of the NAL's resources. On-line searching may 
also be used at-^he discretion of the reference staff in answering 
queries for information received by mall. Scientists and researchers 
outside of DSDA can |et on-linfe to CAIN through Lockheed > Information 
Systems, Palp Alto, California, or Systems Development Corporation, - 



S»ata Monica, California. 
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The Llbyary 

AutoAted Information Retrieval Service 

Most major 'diversity libraries also have a coiqmterlzed llter-^ 
ature searchln^servlce called the Automated Information Retrieval 
Service. The AIRS offers a vide range^of on-line bibliographic and 
statistical data bases* All miijor fields of Inquiiy^re^epresented 
Xn the data bases. ^^Tft^l^^ materials retrieved by the system range 
from technical reports V^V^^^ Information to published articles 
and project descrlptlon^^ro^ Current research activities • Information 
IS' retrieved through <4ceyword 8 or coiid>lnatlon8. of such words appearing 
In titles. In abst^ractSf or In a list of subject terms* Items caxx also 
be recalled by author, by publication name, and/or other categories. 
Output consists q^ig^H st. of citations containing the specified 
combinations of tfiflHls. full bibliographic Information, and in some 
cases 9 abstracts ^^Tprovlded. Host searches cost from, $15 to $30. 
Charges are assessed by the minute according to phone^tlme 
and computer-time. This ^stem contains the Indexes of such information 
syst^^ as: CAIN- Cataloging and Ind^ix^ System of the National 
Agricultural Library; ERIC- Educational. Re^urces Information Center; 
SCISEARCH- Science ClUtlon Index; SQCUiL SCISClRCB- Social Science 
Citation Index; Si^IE- Smltftsonlan Science Information Exchange; and 
TSYCHABS- Psychological Abstracts. To initiate a search, contact a 
4mlv«rsl^ iSbrary^^^ ^ ^ - 



Literature Search ' - « ^ 

For literature searches limited to standard library search 
methods 9 tt^ list of journal abstracts » readers* Indexes » and journals 
belov outlines those soyrces which are most efficient In terms of time 
spent and number of references contained which pertain to evaluative 
research literature, as well as those which were of little use relative ^ 
to this subject matter. 

Xteef^l - Hon-useful 



(a) Sociological Abstracts : parti- 
cularly the Rural Soclolpgy Section 
and abstracts .of papers presented 
at the annual Rural Sociological 
S4(clety; see 1968 and 1971 to 
present. * 

(b) Dissertation Abstracts : in parti- 
cular see, those prior to 1967; once 
dissertations of relevance are lo- 
cated, however, their authors are 
often difficult to lotatelDid ttelr 
manuscripts obtained. See subject 
areaa Valuation*' and "Project." 



(a) Reader *s Guide to Periodical 
Literature 

(b) Wlnchell's Guide to Reference 
Books < J , 

i 

(c) Subject Guide to Books in Print 

(d) Monthly Catalogue of D.S . 
Government Publications ^ ' 
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Useful Non*-U8eful 



(c) Current lnie% to Journals In 
Education (CIJE): an index of 
all manuscripts included in ERIC 
files, as veil a3 laanuscrlpt 
ordering infonnation. 

Cd) Social Science Citation Index : ^ 
a listing of titles and autliors, "-^S^ 
by subject area, etc., which have 
appeared in^ recent social science 
journals. 

i 

(e) Development of Society 

Evaiuatlbn ^ 
Hunan Organization 
Journal of the Coaniunlty 
' Journal of Extension 

Social Issues * 
Sociological Methods & Research 
Social Work * 
Theory and Practice ' 
Transaction r' 
Welfare in Review 



r 



Is Bost useful for locating referencit books ^ 



ta^ogue see 



which may. have extensive bibliographies; in the Subject 
"Evalua|:lon," "Action Research," and "Programs" fii partl^lax. 

Sele<^ted Bibliography 

In this section are selected references which. either deal 
specifically with Rural Development evaluative research and/or offer 
extjgslve bibliographies or special journal Issues pertaining to this 
sujgSct area_. ^ 



Books ^ * • 

Struening,' Elmer and Marcla Guttentag (eds.) 

Handbook of Evaluation Research^ Vol/ 1 & II , Beverly Hills, 

California t Sage Publications, 1975. 

Weiss, Carol (ed.) - 

Evaluating Action Pr ograms; Readings in Social Action and < 
Education . Boston: Allyn and Bacon, 1972. ' 
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Special Journal Issues Devoted to Evaluation 

Sociological' Methods & Research , Vol. 4, No* 2, August. 1973. 
Journal of Extension . No. XIII, March-April, 1975. 

Newsletters 

Evaluation and Rural Development , department of iUiral Sociology, 
> 315 Agriculture Buildings Texas ASM Ui^jLverslty, ^College 

/ Station, Texas 77843. , 

CBp (Commlty Resource DevelBpnent) Newsletter . U. S. Departnent 
of Agriculture, Extension Service, (John S. Bottum, Assistant ♦ 
Administrator, Rural Development Service, USDA, Washington, D.C. 

20250). ^ • • 

V 
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CHAPTER VI 
. EVAmATXOH AMD RDRAI. DEVELOFMEHT* 

b 

Introduction * -# 

Recent guidelines for rural i^rirelopseat (ED) prograas have explicitly 
' Undated evaluation. For exai^le» the gul^^lnea for Title V of the 
Rural DevQlopnent Act. of 1972 require ^prog^B evaluations from agencies 
l^olved In the conduct of the program. This ^of ten places a requirement 
on RD staff ta perform a new function— one In vfalch they usually have 
little training or eicperlince. Altfaou^ a new activity requirement, 
evaluation should not be taken lightly sijuce It provides information 
that may be used \o improve programs; or, in extreme, cases ixlformatlon 
on which the contlnuaiure of programs is decided. In fact, our preferred 
definition of evaluation Is centered on these two purpose. Evaluation 
first tells us how programs wor)fi<aad what §xe Aeix sticcesses and 
failures— an activity! wq thliducan best be achieved throug|i hard-nosed, 
I independent research. Second, evaluati<m seeks to produce information 
1 that can be used tp intelligently discern alternatives for pro-am 
\^ imrovement and change. ' . . . 



Evaluation Models 



Valuation, like any^otber developing, discipline, has adopted a 
particular* jargon. Because It la such a relatively new discipline, 
numerous **models*' exist for evalu^^icm^ Each model is advocated by a 
particular group and tend to repress:^ "the perspectives and biases of 
that group. The term '*mbdel" is soma^t of a.misnmer. Most evalua- 
tors use the terms 'V^els,^ **frameworKb^" and "approaches" 
interchangibly. A more accurate ^escriptbr might be "evaluation 
process" since most "models" depict a prohess whereby programs can be 
evaluated. . ' 

' In our review of Evaluation literatiire, we have Identified more 
than fifty "evaluation models."* They tend tg reflect a variety of 
jargons and emphades; however, it is our opinion that they share certain 
characteristics. He have found by o^^ierlapping the phases of the 
numerous models that most share a set of common, underlying process'^s. 
Recognizing these similarities, a synthetic model has been constructed 
which appears in the following sketch. ^ 



article by Arthur G. Cosby, Associate Professor, Department ^ 
Of Rural Sociology, \Texas Am University and G. Richard Wetherill, 
Research Sociologist, U.S. Forest Service, Southern Forest Experiment 

."^""^^^f^^^"^ 3, Ho. 1 (1977>,>d>llshea 

by the Western Rural pevelopment Center at Corvallis, Oregon 
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Progras 

Descrlptidn 



/ 




Criteria 
FotBuiatlon 



Evidence 
path^ring 

4 




Dlssendnation 



This synthetic model demonstrates the five haslc phases of the 
evaluation process, the first phase/tprogram description) Involves 
docuaentAtlon as to tlie program's £bals and objectives, operation, 
and environment. The second phas^ (criteria forwilat Ion) sets out ^ 
the Qrlt;eria, or accept^bjp^^^ levels of performance, that the program 
is supposed to meet. T^ evid^ide-gathering third, phase is needed 
so that scientific data collectlc^ can be made^ the already- , 
established^cvaluatlvc^ criteria.. 



the data cpllec.ted in the third pha^e are compared, to the crlter^ set 
In the second phase in order to arrive at conelusi^ns regarding program 
successes. and failures. The last phftse in our evaluation model Xs 
the dissemination of finding^ both back Into the program and to 
appropr^te audiences. The dotted arrws in the above Illustration 
indicate that feedback within the model to an earlier phase is possible, 
and in some cases deslralile, when respec if ieat ions in the design are 
deemed necessary. 



Informat ion on Evaluation . ' ' • . 

^ One of -° tte ay r lfld 0f problatt yo * th c; t > €gtffi ttlgg ewltiatorrtB 
Tocating Information on the evaluation enterprise. Lacking scientific 
fora^ (or easily Identifiable) |i||cbanl8M for disses^tlon of ' 4 
evaluation Infomtion^ the bafclc works on evaluative research have 
• been found in the literature of numerous ^cipllajcs. Writers from 
education, public health, «|^logy, public a^plnistratlon, and 
economics have contributed mprle^ge about the fundamentals of 
evaluation.' This implies tBit the evaluator must be familiar with 
diverse sets of literature being produced in disciplines other 
than his own.. 



A ■ore. troublesome inforMtlon difficulty Is the "fugitive" 
.nature of auch evaluation Information. Our reviews have Indicated 
tl>at the vast inajorlty of references, c^speclally thoae reporting the 
actual ^evaluation of programs, appear only as mimeograpbBd reports, 
unpublished papers, and Research monographs. Ihls meaxis^that^ the 
evaluator must also become adept at semmati4§g, Information' In 
order to* keep abreast of the current state-of-ther-art . 

• % ^ 

In addition to the standard library scorch procedures, we have 



found the use of computerized, retrieval systems very helpful— 
especially with unpublished reports. Many agencies and Instltutlpns - 
have this resource available. Ooe benefit here is that someone within 
the agency or Institution is supposed to know hov to use the systems ' 
that are available. Some of the more available retrieval systema 
are the Education Resources Information Centers (ERXC) scattered 
about th^ country. For personnel In rural development » the ERIC/CBESS 
cente* at Hew Mexico State University is probably the most useful 
one*"^' Other systems available are: Smithsonian Science Information 
Exchange (SSIE) ; Current Research Info)iwtion System (CRIS) ;^ Cata- 
loging and I^exlng System of the Natlbm^^vAgrlcultiiral I4brary ^'^ 
(CAm); and the Automated Information Retrll|ial Service OcCRS) avail- 
able throu^ most ma^br university libraries^ ^ 

Perha p s t he best Ind li i Aror of the emferg tng^wirolty of evaluation ^ 
research as a specialized sub-discipline .within th!& 8oc.ial sciences 
Is the recent printing of three major resourceSp In late 1975, Sage 
Publications printed (under the editorship, of Guttentag and Struening) 
tlje first comprehensive work on evaluation The Handbook of Evaluation 
Researj^ . This two-voltnie handbook brin^ together chapters jdealing 
with pfictleally all the issues critical to the conduct of evaluation. 
Scarcely more than a year lat«c^ Sage has undertaken the publication 
of a professional Journal, Evaluation Quarterly , which was the first 
research periodical of its kind directed toward the practice of 
evaluation « At this writing, a second research JourW for evaluation 
(this one from Pergamon Pr^ss) is scheduled for distribution in the 
spring/ Needless to say, these three are indications of a fast-growing 
market for 'evaltitftlon. • — — — " — — 

RecognVzlng that the vast majority of extant literature on 
evaluation and rural development is of 4 rather fugitive nature, a 
network of researchers in the Southern Region (in conJuQetion with 
the Southern Rural Development Center) have been compiling bltbliogta- 
phles in this area. The preliminary bibliography completed in 1976 
contains about ttO referjences with subsections on: major sources, 
definitions of |«raluatlo^^rlteria setting, evaluation strategies, 
respon^lbllltiel-and ethics in evalu^ion, and research-ln-pr ogress. 
An^updated and Imiotated version of uiis blblio^aphy is now available 
fro* the Southern Rural Development Center, Box 5406, Mississippi 
State i Mississippi 39762. While the current supplies last, copies of 
t^he t>rellminary bibliography may be obtained by coijitactlng the Department 
of Rural Socldlogy, Texas AfiM. University, College Station, Texas 77843. 
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Evaluation Training 

Formal evaluation training Is even laore "fugitive" than evaluation 
Information., To our knowledge, there Is a gen^^QT lack of systematic 
training programs that combine a focus on rurai development and program, 
evaluation. If our observations are cotrect, fr^ where will the needed 
expertise come for evaluating RD programs? Most RD evaluation seems to 
be conducted In-house at thls^iiae, %rith project staff -allowjjig only a 
part of their time to the evaluation function. This approach assumes • ' 
that sufficient expertise exists within the project to carry out the 
evaluation requirement. 

Frankly, we are currently falling far short, vof this assume^ level 
,of expertise. Also, we feel that this deficiency is a key contributor 
to the problems that are encountered in evaluating EO) programs. We 
anticipate that this lack of expertise will continue in the future nxnless 
systematic training in evaluation and rural development is initiated in 
key locations aroimd the^tion. 

Pre6entl7^<3the primary educational activity f cfr RD evaluation comes 
from sporadic yorkshops a<k| short courses. We fear that the impetus for 
su^ short-!>tralning courseS results, more often than not, from elevenths- 
bour be eds for evaluation rather than from an Intent to build-in 
evaluation as an integral long-run component of rural development, wltff^ 
consequences of low-^quality, patchwork research. 

The solution to this expertise problem seems, in our mitlds, to be 
based on two types of educational programs. First, there is an obvious 
need fq^^ the development of graduate programs which emphasize both rural 
developKtnt and program evaluation. Currently, the centers for advanced 

-"studies in evaluation (such as those ^t the University of Virginia, 
the University of California—Los Angeles, and the University of 

' Hassachusetts) tend to stress the substantive areas of education and 
health. No parallel training centers have evolved for rural-based 
development programs. It would seem appropriate that our land-grant * 
universities (which h4^ve an explicit emphases on the hinterlands) should 
initiate such graduate training. However, we are not calling for a 
proliferation -of educational programs in evaluation. It is our guess 
that three or four strategically located centers would meet the long- 
run needs of RD evaluation. 

As a second solution^ the im>tivation for, ^nd philosophy underlying 
workshops and short-coursed should be drastically remolded. In this 
short-term solutibt^, program leaders need to emphasize training at the 
earliest possible phase of program 4ev^l(ment. Workshops shoitld be 
systeaatizi^ to Include all facets of ev^^iation". At the state and 
national level, the responsibility sh6uldj>e' accepted, and resources 
allocated, for evaluation well in advance of the need for reports. 
This second strategy is dimply to systematic^ly upgrade in-^house staff. 
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A Progam Dcslg^ and Evialuatl<m Workshop ^ 

' One atteapt to offer aore effective evaluation training within 
the vorkshop format has been sponsored through the Southern Rural 
Developnent Center/ In this workshop, participants from the Southern. 
Region were Isnetsed in an intensive, four-^y program that combined 
program evaluation and rural development. The workshop design began 
with. a general orientation of concepts, designs, and strategies of 
evaluation. From this beginning, participant^ intensely examined the 
methods and techniques of a particular evaluation model (Provus* Discrep-^ 
ancy Evaluation Model) . Then, participant^ applied the Discrepancy 
Evaluation Model tDEH) to their own programs. 

Recently connected with the University pf Virginia's Educational 
Evaluation Center, Dr. Andres Steinmetz (the workshop's director) has 
been working with the educational progra»-oriented DEM in order to adapt 
it to the special problems of evaluating rural development programs. A 
pilot workshop conducted in the Pall of 1976 with South Carolina Title V 
Personnel demonstrated that the DEM training may be an effective vehicle 
for establishing evaluation within rural development. 

The DEM as one strategy for evaluating RD- programs has maxiy selable 
features. ^One of the best features of the model is that the strategy for 
acco^>li8hlng evaluation requires detailed documei|tation of the program 
and its cosqKments. Another unique feature of the model is its apparent 
simplicity of comprehension and utilisation.^ In essence, the DEM 
strategy of evaluation is that of: (1) establishing standards fi^ the 
program, (2) measuring program performance as acco^>llsfaments , snd (3) 
analyzing the discrepancy between standards and performance as an 
evaluative indicator. A net-working language (LOGOS) facilitates the 
program design and evaluation process. ^ 



A Concluding Comt * 

, Although forecasting the future is at best a hazardous enterprise, 
we feel that there are some indications ttat^the demand for quality 
evaluation will dramatically increase in the near future. Our "guessti- 
mation'* rblles heavily on the new administration's expressed preference 
fot zero'^b^sed budget syst»8 of program financing. If zero-based 
budgeting becomes the model method of financing government programs, there 
slimld be a tremendous Increase in requests for rigorous, high credibility 
re9earch that tells which programs are working and how programs can be 
changed to improve their ef fec^veness. Put another way, we expect 
evaluation research to play a significant role in the budgetary process. 
Prom the point-of^view of the evaluation researcher, we suspect the 
Institution of a more formal system of evaluation with less options left 
to the program administrator. We alpo anticipate a much higher demand for 
rigorous evaluation, since credibility of evaluation informaUon with 
policy makers wUl become » critical concern. ^ 
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Announcing 
The Rilral 
Develepmeiit 
Bibliography Soriei 

ten valuable reference books bringing together hundreds of source 



• industriaKzation of rural areas 

• evaluative research' literature 

• citizen participation in rural i 

• solid waste disposal^ financing 
impact of governmental transfer paymenu 



• small farm operations 

• . land use issues ^ 

• health care 

• educational needs projection 
•Jiousing 



A comprehensive p/esentation of recent works in these ppoffty areas of rural development. 
These bibliographiCTs^ill be available as they are printed from the 

Southern Rural Development Center 
Box 5406 

Mississippi State, Mississippi 39762 




Please send me (when avatleble) these volumes: 

Rural Industrialization $5.00 
-_ Citizen Participation 5.00 * '* 

Solid Waste Disposal' 3.00 

Small* Farms "3.00 
Evaluative Liti>rature . 3.00 



^/ 
/ 

— — Lar)d the l^es (to be priced) 

Heattl/Care 

Educational Needs Projection 
Govirnmental Transfer Payments 
HoNUsing 



.enclosed 



. bill me 



ERIC 



Q ^Pleete tend me the entire leriet 

-S2S enclosed — ^_ bill me 
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The^OC it one of four regionaJ rural developnoent 
otntvs in th9 nation, it coordfnftes cooperatipn 
btw f n t»ie Research (Experimiftt Station) and 
Extension (Cooperative Extemion Sarvice) staffs at 
lend-grant institutions in the South to provide tech- 
nical consultation, research, raining, and evalu8ti<^ 
services for rural development This publication is one 
of several published by the Center on various needs, 
program thrusts, «Maseerch efforu in rural develop- 
ment For more information about SRDC activities 
and publications, mite to the Director. ^ 




Soutftkm Rural Oavaiopmant Cantar 
Box 5406 

MiMiwppi.StalB, MS 39762 



The Southsm Rural Developmant Ctntar is an Equal Opportunity Orgsnizrtlon providing rtwerch, educational »n^orm^, and 
otfMT swvicis only to IndWiduali and inttitutlona that function without regard to raca, coior, s^ or national origin. SRDC is an 
Equal Opportunity Employar. 
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